2021-03-09T08:48:37.551 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-09T08:48:37.553 DEBUG:teuthology.run:Teuthology command: teuthology --name kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949099 --description rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/workunits} --verbose -- /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949099/orig.config.yaml 2021-03-09T08:48:37.598 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-09T08:48:37.656 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949099 branch: wip-kefu-testing-2021-03-09-1428 description: rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/workunits} email: tchaikov@gmail.com first_in_suite: false job_id: '5949099' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-kefu-testing-2021-03-09-1428 ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 global: mon election default strategy: 3 mds: debug mgrc: 20 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug mgrc: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2021-03-09-1428 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-03-09-1428 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 targets: smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgPI/Pyly9/2O5ftdKiMCkImpqw/QxFdnpRiJZDbiKE46WaXotGOWYWDu7ZiAdcHvVcSl5Kxyt+JowaqPeWXO80bVcUPOm3joJosEoFH88Gin5vYKNXLr0+m4mNOs1KasAtHRzL7is0XiHWLAyRGWwSLea3y+R5D9R1w5f92vLd1NcYh5YH6dveJu5T+IGXh9ynBGj6iUsahOYxHRDkrSH8oQMOOC+KNKIwNF5h4onoiGYNcTBpZavSQIQ0opJH+SrkwY8eTKNLj83aFQl3uvxgXvOPOGGv36OvPgjENuWsIqo+vf7+5NUxmBFjhjqNe0fgQzVNuOAl7xM/FnALSKJ smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxs8IXgVRubcQLWRZMcAk8SydE9roW7SnHmJFTZrNn9M/flcFLY9fV/utNfnSPvCHwxL1LBSZUplo58pB3rulUtRM0qV6yt1ptwVGm4hRL7QbCFnmAEBgwxfKWNp9LE9j86aU5orVOTVWHoPP25vW38K0LzTyQfhmWsZjPuwvO6QBjP/QwSCJkmaE+/76MX6oxCcjXgmiWXyT6loseF2+Dr34W907z3ocoepvG47ajWNvmeYbDKi5JmcTT1M2Xv60U0bBohcA1v3IwubKH9+jUlU/znWYysPutDGR04umbMmAlVxOhDGx3XBg2MGkUS7zd6vT0LFG68m31E6gQsEKN tasks: - install: null - ceph: log-ignorelist: - overall HEALTH_ - \(MGR_DOWN\) - \(PG_ - replacing it with standby - No standby daemons available wait-for-scrub: false - workunit: clients: client.0: - mgr teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_08:22:40 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 2021-03-09T08:48:37.657 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa; will attempt to use it 2021-03-09T08:48:37.664 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa/tasks 2021-03-09T08:48:37.665 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-09T08:48:37.667 INFO:teuthology.task.internal:Checking packages... 2021-03-09T08:48:37.735 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '89fb622b10ca2e58fe0c913a8956377a886e2ab4' 2021-03-09T08:48:37.735 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-09T08:48:37.736 INFO:teuthology.packaging:ref: None 2021-03-09T08:48:37.736 INFO:teuthology.packaging:tag: None 2021-03-09T08:48:37.736 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-03-09-1428 2021-03-09T08:48:37.737 INFO:teuthology.packaging:sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:48:37.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-kefu-testing-2021-03-09-1428 2021-03-09T08:48:37.897 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:48:37.898 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-09T08:48:37.906 INFO:teuthology.task.internal:no buildpackages task found 2021-03-09T08:48:37.906 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-09T08:48:37.946 INFO:teuthology.task.internal:Saving configuration 2021-03-09T08:48:37.959 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-09T08:48:38.021 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T08:48:38.041 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 08:43:40.614974', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949099', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIITu7ncNYBuG6YV4aFgvhU+s0qz1jbtng6NzKue/kuUv', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi038.front.sepia.ceph.com'} 2021-03-09T08:48:38.059 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 08:43:40.614953', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949099', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEQPishMXR7KfSmB8NIARi/hDc2wXg+YvwPVQ+ApunGJ', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi041.front.sepia.ceph.com'} 2021-03-09T08:48:38.060 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-09T08:48:38.131 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2021-03-09T08:48:38.131 INFO:teuthology.task.internal:roles: ubuntu@smithi041.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2021-03-09T08:48:38.132 INFO:teuthology.run_tasks:Running task console_log... 2021-03-09T08:48:38.229 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8c119cb9d8>, signals=[15]) 2021-03-09T08:48:38.230 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-09T08:48:38.239 INFO:teuthology.task.internal:Opening connections... 2021-03-09T08:48:38.240 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2021-03-09T08:48:38.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:48:38.318 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi041.front.sepia.ceph.com 2021-03-09T08:48:38.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:48:38.388 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-09T08:48:38.398 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2021-03-09T08:48:38.478 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2021-03-09T08:48:38.478 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2021-03-09T08:48:38.525 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="Ubuntu" 2021-03-09T08:48:38.526 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-09T08:48:38.526 INFO:teuthology.orchestra.run.smithi038.stdout:ID=ubuntu 2021-03-09T08:48:38.526 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE=debian 2021-03-09T08:48:38.527 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-09T08:48:38.527 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="18.04" 2021-03-09T08:48:38.527 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T08:48:38.527 INFO:teuthology.orchestra.run.smithi038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T08:48:38.528 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T08:48:38.528 INFO:teuthology.orchestra.run.smithi038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T08:48:38.528 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_CODENAME=bionic 2021-03-09T08:48:38.528 INFO:teuthology.orchestra.run.smithi038.stdout:UBUNTU_CODENAME=bionic 2021-03-09T08:48:38.529 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2021-03-09T08:48:38.546 DEBUG:teuthology.orchestra.run.smithi041:> uname -m 2021-03-09T08:48:38.552 INFO:teuthology.orchestra.run.smithi041.stdout:x86_64 2021-03-09T08:48:38.553 DEBUG:teuthology.orchestra.run.smithi041:> cat /etc/os-release 2021-03-09T08:48:38.599 INFO:teuthology.orchestra.run.smithi041.stdout:NAME="Ubuntu" 2021-03-09T08:48:38.600 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-09T08:48:38.600 INFO:teuthology.orchestra.run.smithi041.stdout:ID=ubuntu 2021-03-09T08:48:38.600 INFO:teuthology.orchestra.run.smithi041.stdout:ID_LIKE=debian 2021-03-09T08:48:38.600 INFO:teuthology.orchestra.run.smithi041.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-09T08:48:38.601 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_ID="18.04" 2021-03-09T08:48:38.601 INFO:teuthology.orchestra.run.smithi041.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T08:48:38.601 INFO:teuthology.orchestra.run.smithi041.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T08:48:38.601 INFO:teuthology.orchestra.run.smithi041.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T08:48:38.602 INFO:teuthology.orchestra.run.smithi041.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T08:48:38.602 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_CODENAME=bionic 2021-03-09T08:48:38.602 INFO:teuthology.orchestra.run.smithi041.stdout:UBUNTU_CODENAME=bionic 2021-03-09T08:48:38.602 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-03-09T08:48:38.618 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-09T08:48:38.631 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-09T08:48:38.684 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-09T08:48:38.685 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2021-03-09T08:48:38.687 DEBUG:teuthology.orchestra.run.smithi041:> test '!' -e /home/ubuntu/cephtest 2021-03-09T08:48:38.693 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-09T08:48:38.770 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-09T08:48:38.771 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2021-03-09T08:48:38.774 DEBUG:teuthology.orchestra.run.smithi041:> test -z $(ls -A /var/lib/ceph) 2021-03-09T08:48:38.782 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-09T08:48:38.875 INFO:teuthology.run_tasks:Running task kernel... 2021-03-09T08:48:38.897 DEBUG:teuthology.task.kernel:normalized config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'mds.c': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mgr.z': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}} 2021-03-09T08:48:38.897 INFO:teuthology.task.kernel:config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-09T08:48:38.898 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2021-03-09T08:48:38.904 INFO:teuthology.orchestra.run.smithi038.stdout:4.15.0-136-generic 2021-03-09T08:48:38.905 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.15.0-136-generic 2021-03-09T08:48:38.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2021-03-09T08:48:39.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2021-03-09T08:48:39.237 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-09T08:48:39.361 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T08:48:39.440 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-09T08:48:39.495 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-09T08:48:39.582 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-09T08:48:39.609 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-09T08:48:39.610 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-09T08:48:39.613 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-09T08:48:39.614 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-09T08:48:39.614 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-09T08:48:39.630 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-09T08:48:39.642 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-09T08:48:39.644 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-09T08:48:39.645 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-09T08:48:39.645 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-09T08:48:39.697 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-09T08:48:39.809 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-09T08:48:40.157 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-09T08:48:40.264 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-09T08:48:40.275 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-09T08:48:40.276 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-09T08:48:40.277 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-09T08:48:40.326 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-09T08:48:40.364 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-09T08:48:40.364 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-09T08:48:40.365 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-09T08:48:41.362 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 12.5 MB in 2s (5,903 kB/s) 2021-03-09T08:48:42.949 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2021-03-09T08:48:42.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T08:48:43.090 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2021-03-09T08:48:43.352 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2021-03-09T08:48:43.352 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2021-03-09T08:48:43.437 INFO:teuthology.orchestra.run.smithi038.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T08:48:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T08:48:43.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2021-03-09T08:48:43.769 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2021-03-09T08:48:43.769 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2021-03-09T08:48:43.988 INFO:teuthology.orchestra.run.smithi038.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-09T08:48:43.988 INFO:teuthology.orchestra.run.smithi038.stdout:linux-image-generic set to manually installed. 2021-03-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T08:48:43.990 DEBUG:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic 2021-03-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic 2021-03-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2021-03-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2021-03-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2021-03-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 17 2021-03-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2021-03-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta 2021-03-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 4.15.0.136.123 2021-03-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2021-03-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2021-03-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2021-03-09T08:48:44.014 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2021-03-09T08:48:44.014 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T08:48:44.015 DEBUG:teuthology.orchestra.run.smithi041:> uname -r 2021-03-09T08:48:44.021 INFO:teuthology.orchestra.run.smithi041.stdout:4.15.0-136-generic 2021-03-09T08:48:44.021 INFO:teuthology.task.kernel:Running kernel on smithi041: 4.15.0-136-generic 2021-03-09T08:48:44.022 DEBUG:teuthology.orchestra.run.smithi041:> sudo apt-get clean 2021-03-09T08:48:44.141 DEBUG:teuthology.orchestra.run.smithi041:> sudo apt-get update 2021-03-09T08:48:44.262 INFO:teuthology.orchestra.run.smithi041.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-09T08:48:44.388 INFO:teuthology.orchestra.run.smithi041.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T08:48:44.469 INFO:teuthology.orchestra.run.smithi041.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-09T08:48:44.519 INFO:teuthology.orchestra.run.smithi041.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-09T08:48:44.610 INFO:teuthology.orchestra.run.smithi041.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-09T08:48:44.640 INFO:teuthology.orchestra.run.smithi041.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-09T08:48:44.641 INFO:teuthology.orchestra.run.smithi041.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-09T08:48:44.644 INFO:teuthology.orchestra.run.smithi041.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-09T08:48:44.644 INFO:teuthology.orchestra.run.smithi041.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-09T08:48:44.645 INFO:teuthology.orchestra.run.smithi041.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-09T08:48:44.657 INFO:teuthology.orchestra.run.smithi041.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-09T08:48:44.666 INFO:teuthology.orchestra.run.smithi041.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-09T08:48:44.669 INFO:teuthology.orchestra.run.smithi041.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-09T08:48:44.669 INFO:teuthology.orchestra.run.smithi041.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-09T08:48:44.669 INFO:teuthology.orchestra.run.smithi041.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-09T08:48:44.735 INFO:teuthology.orchestra.run.smithi041.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-09T08:48:44.855 INFO:teuthology.orchestra.run.smithi041.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-09T08:48:45.214 INFO:teuthology.orchestra.run.smithi041.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-09T08:48:45.317 INFO:teuthology.orchestra.run.smithi041.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-09T08:48:45.318 INFO:teuthology.orchestra.run.smithi041.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-09T08:48:45.330 INFO:teuthology.orchestra.run.smithi041.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-09T08:48:45.330 INFO:teuthology.orchestra.run.smithi041.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-09T08:48:45.378 INFO:teuthology.orchestra.run.smithi041.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-09T08:48:45.418 INFO:teuthology.orchestra.run.smithi041.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-09T08:48:45.421 INFO:teuthology.orchestra.run.smithi041.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-09T08:48:45.421 INFO:teuthology.orchestra.run.smithi041.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-09T08:48:46.296 INFO:teuthology.orchestra.run.smithi041.stdout:Fetched 12.5 MB in 2s (6,360 kB/s) 2021-03-09T08:48:47.820 INFO:teuthology.orchestra.run.smithi041.stdout:Reading package lists... 2021-03-09T08:48:47.870 DEBUG:teuthology.orchestra.run.smithi041:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi041.stdout:Reading package lists... 2021-03-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi041.stdout:Building dependency tree... 2021-03-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi041.stdout:Reading state information... 2021-03-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi041.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T08:48:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T08:48:48.277 DEBUG:teuthology.orchestra.run.smithi041:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi041.stdout:Reading package lists... 2021-03-09T08:48:48.618 INFO:teuthology.orchestra.run.smithi041.stdout:Building dependency tree... 2021-03-09T08:48:48.619 INFO:teuthology.orchestra.run.smithi041.stdout:Reading state information... 2021-03-09T08:48:48.835 INFO:teuthology.orchestra.run.smithi041.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-09T08:48:48.836 INFO:teuthology.orchestra.run.smithi041.stdout:linux-image-generic set to manually installed. 2021-03-09T08:48:48.836 INFO:teuthology.orchestra.run.smithi041.stdout:0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T08:48:48.837 DEBUG:teuthology.orchestra.run.smithi041:> dpkg -s linux-image-generic 2021-03-09T08:48:48.854 INFO:teuthology.orchestra.run.smithi041.stdout:Package: linux-image-generic 2021-03-09T08:48:48.854 INFO:teuthology.orchestra.run.smithi041.stdout:Status: install ok installed 2021-03-09T08:48:48.855 INFO:teuthology.orchestra.run.smithi041.stdout:Priority: optional 2021-03-09T08:48:48.855 INFO:teuthology.orchestra.run.smithi041.stdout:Section: kernel 2021-03-09T08:48:48.855 INFO:teuthology.orchestra.run.smithi041.stdout:Installed-Size: 17 2021-03-09T08:48:48.856 INFO:teuthology.orchestra.run.smithi041.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T08:48:48.856 INFO:teuthology.orchestra.run.smithi041.stdout:Architecture: amd64 2021-03-09T08:48:48.856 INFO:teuthology.orchestra.run.smithi041.stdout:Source: linux-meta 2021-03-09T08:48:48.856 INFO:teuthology.orchestra.run.smithi041.stdout:Version: 4.15.0.136.123 2021-03-09T08:48:48.857 INFO:teuthology.orchestra.run.smithi041.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-09T08:48:48.857 INFO:teuthology.orchestra.run.smithi041.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T08:48:48.857 INFO:teuthology.orchestra.run.smithi041.stdout:Recommends: thermald 2021-03-09T08:48:48.858 INFO:teuthology.orchestra.run.smithi041.stdout:Description: Generic Linux kernel image 2021-03-09T08:48:48.858 INFO:teuthology.orchestra.run.smithi041.stdout: This package will always depend on the latest generic kernel image 2021-03-09T08:48:48.858 INFO:teuthology.orchestra.run.smithi041.stdout: available. 2021-03-09T08:48:48.859 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T08:48:48.859 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2021-03-09T08:48:48.860 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T08:48:48.875 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2021-03-09T08:48:48.893 INFO:teuthology.task.kernel:Enabling kdb on mgr.y... 2021-03-09T08:48:48.893 DEBUG:teuthology.orchestra.run.smithi041:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T08:48:48.915 INFO:teuthology.orchestra.run.smithi041.stdout:ttyS1 2021-03-09T08:48:48.946 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-09T08:48:48.957 INFO:teuthology.task.internal:Creating test directory... 2021-03-09T08:48:48.959 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T08:48:48.961 DEBUG:teuthology.orchestra.run.smithi041:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T08:48:48.993 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-09T08:48:49.112 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-09T08:48:49.157 INFO:teuthology.task.internal:Creating archive directory... 2021-03-09T08:48:49.157 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T08:48:49.160 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T08:48:49.181 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-09T08:48:49.224 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-09T08:48:49.224 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T08:48:49.227 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T08:48:49.250 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:48:49.252 INFO:teuthology.orchestra.run.smithi041.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:48:49.257 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:48:49.260 INFO:teuthology.orchestra.run.smithi041.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:48:49.262 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-09T08:48:49.283 INFO:teuthology.task.internal:Configuring sudo... 2021-03-09T08:48:49.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T08:48:49.304 DEBUG:teuthology.orchestra.run.smithi041:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T08:48:49.321 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-09T08:48:49.345 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-09T08:48:49.346 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T08:48:49.360 DEBUG:teuthology.orchestra.run.smithi041:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T08:48:49.370 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:48:49.413 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:48:49.465 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:48:49.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T08:48:49.532 DEBUG:teuthology.orchestra.run.smithi041:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:48:49.539 DEBUG:teuthology.orchestra.run.smithi041:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:48:49.590 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:48:49.590 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T08:48:49.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2021-03-09T08:48:49.658 DEBUG:teuthology.orchestra.run.smithi041:> sudo service rsyslog restart 2021-03-09T08:48:49.736 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-09T08:48:49.746 INFO:teuthology.task.internal:Starting timer... 2021-03-09T08:48:49.747 INFO:teuthology.run_tasks:Running task pcp... 2021-03-09T08:48:49.840 INFO:teuthology.run_tasks:Running task selinux... 2021-03-09T08:48:49.907 DEBUG:teuthology.task.selinux:Excluding smithi038: OS 'ubuntu' does not support SELinux 2021-03-09T08:48:49.907 DEBUG:teuthology.task.selinux:Excluding smithi041: OS 'ubuntu' does not support SELinux 2021-03-09T08:48:49.908 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-09T08:48:49.908 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-09T08:48:49.908 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-09T08:48:49.909 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-09T08:48:49.970 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-03-09T08:48:49.982 INFO:teuthology.repo_utils:Fetching master from origin 2021-03-09T08:48:50.177 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-09T08:48:50.401 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-09T08:48:50.409 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi041.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-09T08:48:51.146 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-09T08:48:52.734 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-09T08:48:54.987 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-09T08:48:55.046 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:55 +0000 (0:00:00.088) 0:00:00.088 ********* 2021-03-09T08:48:55.064 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:55 +0000 (0:00:00.017) 0:00:00.106 ********* 2021-03-09T08:48:56.704 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T08:48:56.706 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi038.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi038.front.sepia.ceph.com] 2021-03-09T08:48:56.757 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:56 +0000 (0:00:01.693) 0:00:01.799 ********* 2021-03-09T08:48:56.840 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi041.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi041.front.sepia.ceph.com] 2021-03-09T08:48:56.890 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:56 +0000 (0:00:00.133) 0:00:01.932 ********* 2021-03-09T08:48:57.020 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-09T08:48:57.022 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:48:57.043 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:57 +0000 (0:00:00.152) 0:00:02.085 ********* 2021-03-09T08:48:57.061 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:57 +0000 (0:00:00.018) 0:00:02.103 ********* 2021-03-09T08:48:57.687 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-09T08:48:57.689 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T08:48:57.712 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:57 +0000 (0:00:00.650) 0:00:02.754 ********* 2021-03-09T08:48:57.719 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T08:48:57.747 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:57 +0000 (0:00:00.034) 0:00:02.789 ********* 2021-03-09T08:48:58.074 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-09T08:48:58.075 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.068810", "end": "2021-03-09 08:48:57.789868", "rc": 0, "start": "2021-03-09 08:48:57.721058", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.068426", "end": "2021-03-09 08:48:58.093821", "rc": 0, "start": "2021-03-09 08:48:58.025395", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-09T08:48:58.097 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:58 +0000 (0:00:00.349) 0:00:03.139 ********* 2021-03-09T08:48:58.116 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:58 +0000 (0:00:00.018) 0:00:03.157 ********* 2021-03-09T08:48:58.358 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-03-09T08:48:58.360 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:48:58.386 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:58 +0000 (0:00:00.270) 0:00:03.428 ********* 2021-03-09T08:48:58.417 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:58 +0000 (0:00:00.030) 0:00:03.459 ********* 2021-03-09T08:48:59.255 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-03-09T08:48:59.257 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279738.450782-6202-265088157453414/source", "state": "file", "uid": 0} changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279738.428335-6200-253074138983497/source", "state": "file", "uid": 0} 2021-03-09T08:48:59.279 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:59 +0000 (0:00:00.862) 0:00:04.321 ********* 2021-03-09T08:48:59.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:48:59 +0000 (0:00:00.019) 0:00:04.341 ********* 2021-03-09T08:48:59.750 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-09T08:48:59.757 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T08:48:59.759 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T08:48:59.774 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T08:48:59.776 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T08:48:59.852 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-09T08:48:59.856 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-09T08:49:00.021 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:00 +0000 (0:00:00.721) 0:00:05.062 ********* 2021-03-09T08:49:00.042 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:00 +0000 (0:00:00.020) 0:00:05.083 ********* 2021-03-09T08:49:00.721 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi038.front.sepia.ceph.com] 2021-03-09T08:49:00.777 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-09T08:49:00.778 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:00 +0000 (0:00:00.735) 0:00:05.819 ********* 2021-03-09T08:49:00.787 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] 2021-03-09T08:49:00.840 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:00 +0000 (0:00:00.063) 0:00:05.882 ********* 2021-03-09T08:49:01.654 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-09T08:49:01.657 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.494087", "end": "2021-03-09 08:49:01.623919", "rc": 0, "start": "2021-03-09 08:49:01.129832", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T08:49:01.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:01 +0000 (0:00:00.839) 0:00:06.722 ********* 2021-03-09T08:49:01.698 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.546370", "end": "2021-03-09 08:49:01.676006", "rc": 0, "start": "2021-03-09 08:49:01.129636", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T08:49:01.728 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:01 +0000 (0:00:00.047) 0:00:06.770 ********* 2021-03-09T08:49:01.852 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T08:49:01.854 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014671", "end": "2021-03-09 08:49:01.869604", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:49:01.854933", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T08:49:01.880 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:01 +0000 (0:00:00.152) 0:00:06.922 ********* 2021-03-09T08:49:01.887 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014983", "end": "2021-03-09 08:49:01.595031", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:49:01.580048", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T08:49:01.917 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:01 +0000 (0:00:00.036) 0:00:06.959 ********* 2021-03-09T08:49:01.966 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T08:49:01.968 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:01.991 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:01 +0000 (0:00:00.073) 0:00:07.033 ********* 2021-03-09T08:49:02.008 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.032 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.041) 0:00:07.074 ********* 2021-03-09T08:49:02.059 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-09T08:49:02.061 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.086 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.053) 0:00:07.128 ********* 2021-03-09T08:49:02.123 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.148 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.062) 0:00:07.190 ********* 2021-03-09T08:49:02.154 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-09T08:49:02.155 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.180 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.032) 0:00:07.222 ********* 2021-03-09T08:49:02.229 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.253 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.072) 0:00:07.295 ********* 2021-03-09T08:49:02.259 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T08:49:02.261 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.284 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.031) 0:00:07.326 ********* 2021-03-09T08:49:02.333 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.359 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.074) 0:00:07.401 ********* 2021-03-09T08:49:02.376 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-09T08:49:02.378 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.040) 0:00:07.442 ********* 2021-03-09T08:49:02.427 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:02.451 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.051) 0:00:07.493 ********* 2021-03-09T08:49:02.593 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-09T08:49:02.595 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:49:02.617 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.165) 0:00:07.658 ********* 2021-03-09T08:49:02.647 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:49:02.686 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:02 +0000 (0:00:00.069) 0:00:07.728 ********* 2021-03-09T08:49:02.894 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:49:02.896 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T08:49:02.939 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T08:49:03.056 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T08:49:03.088 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T08:49:03.226 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T08:49:03.250 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T08:49:03.367 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T08:49:03.400 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T08:49:03.535 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T08:49:03.547 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T08:49:03.683 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T08:49:03.694 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T08:49:03.830 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T08:49:03.852 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T08:49:03.977 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T08:49:04.020 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T08:49:04.125 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T08:49:04.178 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T08:49:04.272 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T08:49:04.346 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T08:49:04.420 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T08:49:04.503 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T08:49:04.566 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T08:49:04.660 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T08:49:04.713 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T08:49:04.797 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T08:49:04.860 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T08:49:04.965 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T08:49:05.008 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T08:49:05.133 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T08:49:05.155 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T08:49:05.291 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T08:49:05.302 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T08:49:05.448 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T08:49:05.460 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T08:49:05.586 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T08:49:05.598 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T08:49:05.733 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T08:49:05.755 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T08:49:05.880 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T08:49:05.912 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T08:49:06.027 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T08:49:06.071 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T08:49:06.176 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T08:49:06.228 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T08:49:06.313 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T08:49:06.397 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T08:49:06.460 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T08:49:06.543 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T08:49:06.607 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T08:49:06.712 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T08:49:06.754 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T08:49:06.869 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T08:49:06.901 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T08:49:07.026 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T08:49:07.048 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T08:49:07.183 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T08:49:07.195 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T08:49:07.330 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T08:49:07.352 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T08:49:07.477 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T08:49:07.499 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T08:49:07.624 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T08:49:07.656 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T08:49:07.771 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T08:49:07.824 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T08:49:07.918 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T08:49:07.984 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T08:49:08.058 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T08:49:08.146 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T08:49:08.209 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T08:49:08.314 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T08:49:08.356 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T08:49:08.461 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T08:49:08.503 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:49:08.628 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T08:49:08.640 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T08:49:08.785 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T08:49:08.786 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T08:49:08.932 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:49:08.933 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T08:49:09.078 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T08:49:09.100 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T08:49:09.225 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T08:49:09.257 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T08:49:09.373 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T08:49:09.406 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T08:49:09.521 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T08:49:09.542 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T08:49:09.657 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:49:09.689 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T08:49:09.804 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T08:49:09.857 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T08:49:09.951 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T08:49:10.004 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T08:49:10.098 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T08:49:10.140 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:49:10.234 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T08:49:10.308 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T08:49:10.381 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T08:49:10.475 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T08:49:10.517 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T08:49:10.632 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T08:49:10.664 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T08:49:10.799 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T08:49:10.800 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T08:49:10.946 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T08:49:10.968 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T08:49:11.083 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T08:49:11.115 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T08:49:11.229 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T08:49:11.272 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T08:49:11.376 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T08:49:11.439 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T08:49:11.513 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T08:49:11.607 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T08:49:11.660 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T08:49:11.774 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T08:49:11.806 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T08:49:11.921 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T08:49:11.954 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T08:49:12.089 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T08:49:12.092 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:49:12.236 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T08:49:12.258 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T08:49:12.383 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T08:49:12.426 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T08:49:12.530 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T08:49:12.582 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T08:49:12.677 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T08:49:12.752 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T08:49:12.817 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T08:49:12.891 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:49:12.966 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T08:49:13.052 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T08:49:13.106 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T08:49:13.221 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T08:49:13.253 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T08:49:13.389 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T08:49:13.391 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T08:49:13.537 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T08:49:13.559 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T08:49:13.684 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T08:49:13.717 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T08:49:13.833 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T08:49:13.879 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T08:49:13.974 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T08:49:14.027 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T08:49:14.111 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T08:49:14.184 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T08:49:14.258 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T08:49:14.342 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T08:49:14.405 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T08:49:14.488 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T08:49:14.551 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T08:49:14.635 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T08:49:14.698 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T08:49:14.794 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T08:49:14.835 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T08:49:14.960 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T08:49:14.983 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T08:49:15.122 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T08:49:15.123 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T08:49:15.271 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T08:49:15.272 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T08:49:15.418 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T08:49:15.429 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T08:49:15.564 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T08:49:15.597 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T08:49:15.711 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T08:49:15.754 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T08:49:15.848 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T08:49:15.901 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T08:49:15.995 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T08:49:16.069 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T08:49:16.132 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T08:49:16.205 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T08:49:16.279 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T08:49:16.373 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T08:49:16.415 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T08:49:16.540 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T08:49:16.561 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T08:49:16.707 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T08:49:16.707 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T08:49:16.852 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T08:49:16.874 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T08:49:16.999 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T08:49:17.041 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T08:49:17.135 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T08:49:17.199 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T08:49:17.282 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T08:49:17.366 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T08:49:17.428 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T08:49:17.533 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T08:49:17.575 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:49:17.691 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T08:49:17.723 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T08:49:17.859 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T08:49:17.870 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T08:49:18.021 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T08:49:18.023 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T08:49:18.179 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T08:49:18.325 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T08:49:18.491 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T08:49:18.648 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T08:49:18.814 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T08:49:18.971 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T08:49:18.993 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:49:19.119 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T08:49:19.162 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T08:49:19.267 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T08:49:19.329 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T08:49:19.413 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T08:49:19.486 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T08:49:19.560 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T08:49:19.643 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T08:49:19.696 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T08:49:19.801 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T08:49:19.843 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T08:49:19.948 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T08:49:19.991 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T08:49:20.096 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T08:49:20.150 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T08:49:20.244 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T08:49:20.286 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T08:49:20.412 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T08:49:20.434 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T08:49:20.569 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T08:49:20.580 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T08:49:20.716 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T08:49:20.728 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T08:49:20.867 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T08:49:20.879 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T08:49:21.004 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T08:49:21.026 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T08:49:21.151 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T08:49:21.173 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T08:49:21.298 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T08:49:21.320 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T08:49:21.445 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T08:49:21.488 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T08:49:21.593 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T08:49:21.646 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T08:49:21.740 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T08:49:21.814 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T08:49:21.888 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T08:49:21.961 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T08:49:22.034 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T08:49:22.118 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T08:49:22.181 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T08:49:22.265 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T08:49:22.318 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T08:49:22.414 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T08:49:22.466 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T08:49:22.581 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T08:49:22.613 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T08:49:22.748 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T08:49:22.760 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T08:49:22.895 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T08:49:22.907 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T08:49:23.044 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T08:49:23.056 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T08:49:23.203 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T08:49:23.328 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T08:49:23.360 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T08:49:23.464 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T08:49:23.527 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T08:49:23.611 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T08:49:23.664 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T08:49:23.759 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T08:49:23.833 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T08:49:23.916 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T08:49:24.017 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T08:49:24.053 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:24 +0000 (0:00:21.367) 0:00:29.095 ********* 2021-03-09T08:49:24.136 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T08:49:24.161 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T08:49:24.183 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:24 +0000 (0:00:00.130) 0:00:29.225 ********* 2021-03-09T08:49:24.282 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:49:24.284 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T08:49:24.410 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T08:49:24.412 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T08:49:24.413 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T08:49:24.415 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T08:49:24.417 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T08:49:24.418 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRah 2021-03-09T08:49:24.419 INFO:teuthology.task.ansible.out:w fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T08:49:24.420 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T08:49:24.422 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T08:49:24.424 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T08:49:24.425 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T08:49:24.427 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T08:49:24.429 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T08:49:24.430 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-noteboo 2021-03-09T08:49:24.431 INFO:teuthology.task.ansible.out:k LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T08:49:24.432 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T08:49:24.458 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:24 +0000 (0:00:00.275) 0:00:29.500 ********* 2021-03-09T08:49:24.465 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:49:24.466 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T08:49:24.596 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T08:49:24.899 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T08:49:25.121 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:25 +0000 (0:00:00.662) 0:00:30.163 ********* 2021-03-09T08:49:25.191 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T08:49:25.214 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:25 +0000 (0:00:00.092) 0:00:30.255 ********* 2021-03-09T08:49:25.304 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T08:49:25.306 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T08:49:25.308 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T08:49:25.309 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T08:49:25.311 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\ 2021-03-09T08:49:25.312 INFO:teuthology.task.ansible.out:nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T08:49:25.314 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T08:49:25.316 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T08:49:25.317 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T08:49:25.319 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T08:49:25.320 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshi 2021-03-09T08:49:25.321 INFO:teuthology.task.ansible.out:t@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T08:49:25.322 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T08:49:25.324 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T08:49:25.326 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T08:49:25.328 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, 2021-03-09T08:49:25.328 INFO:teuthology.task.ansible.out: {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T08:49:25.357 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:25 +0000 (0:00:00.142) 0:00:30.398 ********* 2021-03-09T08:49:25.861 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-09T08:49:25.863 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T08:49:25.889 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:25 +0000 (0:00:00.532) 0:00:30.931 ********* 2021-03-09T08:49:25.993 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T08:49:26.018 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:26 +0000 (0:00:00.128) 0:00:31.060 ********* 2021-03-09T08:49:26.138 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:49:26.140 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T08:49:26.253 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T08:49:26.275 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T08:49:26.404 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T08:49:26.405 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T08:49:26.539 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T08:49:26.541 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T08:49:26.550 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T08:49:26.675 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T08:49:26.707 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T08:49:26.709 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T08:49:26.811 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T08:49:26.813 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T08:49:26.853 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T08:49:26.947 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T08:49:27.000 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T08:49:27.002 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T08:49:27.094 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T08:49:27.149 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T08:49:27.273 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T08:49:27.276 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhb 2021-03-09T08:49:27.279 INFO:teuthology.task.ansible.out:l1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:27.412 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T08:49:27.413 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T08:49:27.423 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T08:49:27.434 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:27.548 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T08:49:27.550 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T08:49:27.560 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:27.561 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:27.571 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T08:49:27.573 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T08:49:27.697 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T08:49:27.698 INFO:teuthology.task.ansible.out:son", "validate_certs": true} ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33J 2021-03-09T08:49:27.700 INFO:teuthology.task.ansible.out:iv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:27.709 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:27.823 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T08:49:27.825 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T08:49:27.834 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T08:49:27.960 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T08:49:27.962 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T08:49:27.964 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T08:49:28.101 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T08:49:28.104 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T08:49:28.227 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T08:49:28.229 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-09T08:49:28.249 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T08:49:28.364 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T08:49:28.376 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T08:49:28.512 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T08:49:28.513 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T08:49:28.649 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T08:49:28.651 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9 2021-03-09T08:49:28.653 INFO:teuthology.task.ansible.out:AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T08:49:28.787 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T08:49:28.789 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T08:49:28.798 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T08:49:28.800 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T08:49:28.923 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T08:49:28.925 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T08:49:29.049 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T08:49:29.060 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T08:49:29.185 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T08:49:29.196 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T08:49:29.198 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T08:49:29.321 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T08:49:29.323 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T08:49:29.333 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T08:49:29.457 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T08:49:29.479 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T08:49:29.593 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T08:49:29.615 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T08:49:29.720 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T08:49:29.752 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T08:49:29.857 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T08:49:29.899 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T08:49:29.983 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T08:49:29.984 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T08:49:30.038 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T08:49:30.121 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T08:49:30.163 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T08:49:30.165 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T08:49:30.257 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T08:49:30.289 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T08:49:30.393 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T08:49:30.435 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T08:49:30.529 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T08:49:30.571 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T08:49:30.666 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T08:49:30.718 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T08:49:30.802 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T08:49:30.854 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T08:49:30.928 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:49:31.012 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T08:49:31.064 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T08:49:31.148 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:49:31.201 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T08:49:31.287 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T08:49:31.330 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T08:49:31.332 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T08:49:31.416 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T08:49:31.470 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T08:49:31.472 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T08:49:31.554 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T08:49:31.556 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T08:49:31.607 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T08:49:31.701 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T08:49:31.703 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T08:49:31.733 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T08:49:31.837 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T08:49:31.869 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T08:49:31.974 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T08:49:32.012 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T08:49:32.014 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:49:32.121 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T08:49:32.144 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T08:49:32.150 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T08:49:32.152 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T08:49:32.270 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T08:49:32.272 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:49:32.313 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T08:49:32.428 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T08:49:32.430 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T08:49:32.431 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T08:49:32.480 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T08:49:32.564 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T08:49:32.617 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T08:49:32.734 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T08:49:32.766 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T08:49:32.892 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T08:49:32.923 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T08:49:33.060 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T08:49:33.082 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T08:49:33.084 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T08:49:33.197 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T08:49:33.240 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T08:49:33.346 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T08:49:33.347 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T08:49:33.379 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T08:49:33.484 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T08:49:33.528 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T08:49:33.530 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T08:49:33.645 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T08:49:33.666 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T08:49:33.782 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T08:49:33.783 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T08:49:33.803 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T08:49:33.929 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T08:49:33.930 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_k 2021-03-09T08:49:33.932 INFO:teuthology.task.ansible.out:eys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T08:49:34.055 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T08:49:34.066 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T08:49:34.203 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T08:49:34.204 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T08:49:34.206 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T08:49:34.340 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T08:49:34.361 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T08:49:34.466 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:49:34.508 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T08:49:34.602 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T08:49:34.655 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T08:49:34.739 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T08:49:34.741 INFO:teuthology.task.ansible.out: 2021-03-09T08:49:34.782 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:49:34.887 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T08:49:34.930 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T08:49:35.014 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T08:49:35.077 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T08:49:35.151 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T08:49:35.214 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T08:49:35.288 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T08:49:35.361 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T08:49:35.424 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T08:49:35.498 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T08:49:35.550 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T08:49:35.655 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T08:49:35.687 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T08:49:35.791 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T08:49:35.844 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T08:49:35.928 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T08:49:35.980 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T08:49:36.064 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T08:49:36.116 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T08:49:36.200 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T08:49:36.242 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T08:49:36.347 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T08:49:36.379 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T08:49:36.483 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T08:49:36.505 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T08:49:36.609 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T08:49:36.641 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T08:49:36.746 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T08:49:36.778 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T08:49:36.780 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T08:49:36.872 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T08:49:36.914 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T08:49:37.029 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T08:49:37.050 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T08:49:37.175 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T08:49:37.177 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T08:49:37.186 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T08:49:37.312 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T08:49:37.323 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T08:49:37.448 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T08:49:37.450 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T08:49:37.459 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T08:49:37.585 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T08:49:37.586 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T08:49:37.617 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T08:49:37.721 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T08:49:37.764 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T08:49:37.857 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T08:49:37.921 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T08:49:37.922 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T08:49:37.994 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T08:49:38.078 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T08:49:38.081 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T08:49:38.131 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T08:49:38.194 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T08:49:38.267 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T08:49:38.331 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T08:49:38.404 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T08:49:38.457 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T08:49:38.541 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T08:49:38.604 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T08:49:38.690 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T08:49:38.692 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T08:49:38.753 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T08:49:38.816 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T08:49:38.900 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T08:49:38.953 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T08:49:39.036 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T08:49:39.089 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T08:49:39.183 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T08:49:39.185 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T08:49:39.225 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T08:49:39.320 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T08:49:39.352 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T08:49:39.467 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T08:49:39.489 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T08:49:39.614 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T08:49:39.625 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T08:49:39.627 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:49:39.750 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T08:49:39.761 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T08:49:39.886 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T08:49:39.897 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T08:49:40.022 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T08:49:40.033 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T08:49:40.159 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T08:49:40.180 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T08:49:40.182 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:49:40.296 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T08:49:40.318 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T08:49:40.433 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T08:49:40.475 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T08:49:40.569 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T08:49:40.612 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T08:49:40.706 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T08:49:40.758 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T08:49:40.842 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T08:49:40.905 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T08:49:40.978 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T08:49:41.052 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T08:49:41.104 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T08:49:41.198 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T08:49:41.241 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T08:49:41.335 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T08:49:41.377 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T08:49:41.482 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T08:49:41.503 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T08:49:41.629 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T08:49:41.640 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T08:49:41.775 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T08:49:41.787 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T08:49:41.904 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T08:49:41.926 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T08:49:42.031 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T08:49:42.052 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T08:49:42.177 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T08:49:42.189 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T08:49:42.191 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T08:49:42.303 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T08:49:42.315 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T08:49:42.451 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T08:49:42.453 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T08:49:42.598 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T08:49:42.600 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T08:49:42.734 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T08:49:42.736 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T08:49:42.860 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T08:49:42.883 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T08:49:42.885 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T08:49:42.999 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T08:49:43.011 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T08:49:43.137 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T08:49:43.158 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T08:49:43.294 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T08:49:43.315 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T08:49:43.420 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T08:49:43.442 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T08:49:43.558 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T08:49:43.569 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T08:49:43.689 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T08:49:43.721 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T08:49:43.825 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T08:49:43.857 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T08:49:43.962 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T08:49:43.994 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T08:49:44.109 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T08:49:44.152 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T08:49:44.246 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T08:49:44.278 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T08:49:44.393 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T08:49:44.414 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T08:49:44.531 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T08:49:44.533 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T08:49:44.564 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T08:49:44.670 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T08:49:44.742 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T08:49:44.777 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:44 +0000 (0:00:18.758) 0:00:49.818 ********* 2021-03-09T08:49:44.852 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T08:49:44.917 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:49:44.918 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.928 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.940 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.951 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.953 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.965 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.976 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.987 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.989 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:44.999 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:49:45.001 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T08:49:45.127 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T08:49:45.148 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:49:45.150 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T08:49:45.160 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.162 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.256 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:49:45.258 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T08:49:45.259 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T08:49:45.299 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:49:45.301 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T08:49:45.407 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:49:45.409 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T08:49:45.479 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:49:45.481 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-03-09T08:49:45.482 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@n 2021-03-09T08:49:45.484 INFO:teuthology.task.ansible.out:ew-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2 2021-03-09T08:49:45.486 INFO:teuthology.task.ansible.out:b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.508 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:45 +0000 (0:00:00.731) 0:00:50.550 ********* 2021-03-09T08:49:45.516 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.517 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.518 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.532 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.533 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.544 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.554 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.556 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.566 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.577 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.588 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.589 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.599 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.601 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.610 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.612 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.623 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.625 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.635 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.639 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.648 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.650 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.661 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.663 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.674 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.676 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.687 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.699 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.711 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.723 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.724 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.726 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.737 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.749 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.751 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.763 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.764 INFO:teuthology.task.ansible.out: skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.776 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.788 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.790 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.800 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.802 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T08:49:45.803 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.815 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.818 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.818 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@e 2021-03-09T08:49:45.819 INFO:teuthology.task.ansible.out:lectric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.829 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.831 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.841 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.842 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.854 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.855 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.868 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.868 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.880 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.882 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.894 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.895 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.907 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.908 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.920 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.920 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.932 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.943 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.944 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.954 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.956 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhn 2021-03-09T08:49:45.958 INFO:teuthology.task.ansible.out:S0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T08:49:45.967 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.969 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.979 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.990 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:45.992 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.002 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.003 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.013 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.024 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.026 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.036 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.038 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.047 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.049 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.058 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.069 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.071 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.081 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.082 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.092 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.103 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.104 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+ 2021-03-09T08:49:46.106 INFO:teuthology.task.ansible.out:i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T08:49:46.118 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.127 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.129 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.131 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.142 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.144 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.154 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.156 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.167 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.169 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.169 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.179 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.181 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.192 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.194 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.204 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.206 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.207 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.218 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.220 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.231 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.233 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.244 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.246 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.246 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.247 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.258 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.259 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.270 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.272 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.283 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.284 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.285 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.296 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.298 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.308 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.309 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.320 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.321 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.332 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.334 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.351 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.353 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.376 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:46 +0000 (0:00:00.868) 0:00:51.418 ********* 2021-03-09T08:49:46.383 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.384 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.386 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.399 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.401 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.414 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.415 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.427 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.429 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.441 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.443 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.453 INFO:teuthology.task.ansible.out: 2021-03-09T08:49:46.455 INFO:teuthology.task.ansible.out:TASK [Filter the revoked_users list] ******************************************* skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.481 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:46 +0000 (0:00:00.104) 0:00:51.522 ********* 2021-03-09T08:49:46.488 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.491 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.491 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.503 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.514 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.516 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.526 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.528 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.537 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.548 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.550 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.560 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.571 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.582 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.584 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.594 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.596 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.605 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.617 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.618 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.629 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.641 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.642 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.653 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.664 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.675 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.677 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.688 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.690 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.700 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.702 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.715 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.717 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi041.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.729 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.729 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.741 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.743 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.755 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.756 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.767 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.769 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.779 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.779 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.790 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.802 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.814 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.815 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.826 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.828 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.838 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.840 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.851 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.853 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.862 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:46.865 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.875 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.877 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.887 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.888 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.899 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.900 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.910 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.921 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.923 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.933 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.934 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.944 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.946 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.956 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.958 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.968 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:46.969 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:47.019 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:47 +0000 (0:00:00.538) 0:00:52.061 ********* 2021-03-09T08:49:47.025 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:47.027 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T08:49:47.091 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:47.119 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:47 +0000 (0:00:00.099) 0:00:52.160 ********* 2021-03-09T08:49:47.157 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:47.158 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T08:49:47.293 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T08:49:47.335 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T08:49:47.430 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T08:49:47.483 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T08:49:47.577 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T08:49:47.630 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T08:49:47.713 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T08:49:47.789 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T08:49:47.852 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T08:49:47.936 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T08:49:47.999 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T08:49:48.083 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T08:49:48.136 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T08:49:48.229 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T08:49:48.282 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T08:49:48.376 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T08:49:48.430 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T08:49:48.513 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T08:49:48.567 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T08:49:48.662 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T08:49:48.716 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T08:49:48.831 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T08:49:48.853 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T08:49:48.978 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T08:49:48.990 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T08:49:49.125 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T08:49:49.137 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T08:49:49.272 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T08:49:49.274 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T08:49:49.410 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T08:49:49.422 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T08:49:49.548 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T08:49:49.560 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T08:49:49.687 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T08:49:49.719 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T08:49:49.826 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T08:49:49.870 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T08:49:49.974 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T08:49:50.018 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T08:49:50.113 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T08:49:50.167 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T08:49:50.252 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T08:49:50.316 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T08:49:50.399 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T08:49:50.463 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T08:49:50.536 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T08:49:50.610 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T08:49:50.683 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T08:49:50.757 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T08:49:50.811 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T08:49:50.926 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T08:49:50.960 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T08:49:51.055 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T08:49:51.098 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T08:49:51.213 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T08:49:51.234 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T08:49:51.360 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T08:49:51.372 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T08:49:51.511 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T08:49:51.512 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T08:49:51.648 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T08:49:51.661 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T08:49:51.797 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T08:49:51.809 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T08:49:51.934 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T08:49:51.956 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T08:49:52.082 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T08:49:52.104 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T08:49:52.221 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T08:49:52.243 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T08:49:52.359 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T08:49:52.391 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T08:49:52.497 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T08:49:52.529 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T08:49:52.634 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T08:49:52.676 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T08:49:52.771 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T08:49:52.824 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T08:49:52.907 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T08:49:52.970 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T08:49:53.054 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T08:49:53.118 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T08:49:53.192 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T08:49:53.255 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T08:49:53.329 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T08:49:53.413 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T08:49:53.476 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T08:49:53.530 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:53 +0000 (0:00:06.411) 0:00:58.572 ********* 2021-03-09T08:49:53.548 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T08:49:53.704 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T08:49:53.851 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T08:49:53.955 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T08:49:53.957 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279793.6285076-8072-79636686542885/source", "state": "file", "uid": 0} 2021-03-09T08:49:53.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:53 +0000 (0:00:00.450) 0:00:59.022 ********* 2021-03-09T08:49:53.997 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:53.999 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T08:49:54.062 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T08:49:54.064 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => { "msg": "Host smithi041.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-09T08:49:54.087 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.106) 0:00:59.128 ********* 2021-03-09T08:49:54.156 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:54.158 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T08:49:54.178 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T08:49:54.183 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T08:49:54.208 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.121) 0:00:59.250 ********* 2021-03-09T08:49:54.276 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-09T08:49:54.278 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.091) 0:00:59.341 ********* 2021-03-09T08:49:54.305 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:54.307 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T08:49:54.370 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T08:49:54.371 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.393 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.094) 0:00:59.435 ********* 2021-03-09T08:49:54.461 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:49:54.463 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T08:49:54.484 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T08:49:54.485 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.507 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.113) 0:00:59.549 ********* 2021-03-09T08:49:54.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.036) 0:00:59.586 ********* 2021-03-09T08:49:54.620 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T08:49:54.622 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.650 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.105) 0:00:59.692 ********* 2021-03-09T08:49:54.730 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T08:49:54.732 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.756 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.105) 0:00:59.798 ********* 2021-03-09T08:49:54.825 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T08:49:54.827 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.849 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.093) 0:00:59.891 ********* 2021-03-09T08:49:54.919 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T08:49:54.920 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:54.944 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.094) 0:00:59.986 ********* 2021-03-09T08:49:54.961 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T08:49:54.963 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279794.6132917-8121-156783145737004/source", "state": "file", "uid": 0} 2021-03-09T08:49:54.991 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:54 +0000 (0:00:00.046) 0:01:00.032 ********* 2021-03-09T08:49:55.009 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T08:49:55.011 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.036 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.045) 0:01:00.077 ********* 2021-03-09T08:49:55.077 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T08:49:55.078 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => { "msg": "Host smithi038.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-09T08:49:55.103 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.067) 0:01:00.144 ********* 2021-03-09T08:49:55.120 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T08:49:55.122 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:55.145 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.042) 0:01:00.187 ********* 2021-03-09T08:49:55.184 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T08:49:55.185 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T08:49:55.210 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.064) 0:01:00.252 ********* 2021-03-09T08:49:55.228 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-09T08:49:55.230 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.257 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.046) 0:01:00.299 ********* 2021-03-09T08:49:55.285 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-09T08:49:55.287 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.312 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.054) 0:01:00.354 ********* 2021-03-09T08:49:55.340 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T08:49:55.342 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.053) 0:01:00.407 ********* 2021-03-09T08:49:55.382 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T08:49:55.384 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.410 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.044) 0:01:00.452 ********* 2021-03-09T08:49:55.449 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T08:49:55.451 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:55.477 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.067) 0:01:00.519 ********* 2021-03-09T08:49:55.495 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T08:49:55.495 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.522 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.044) 0:01:00.564 ********* 2021-03-09T08:49:55.524 INFO:teuthology.task.ansible.out: 2021-03-09T08:49:55.551 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T08:49:55.553 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.054) 0:01:00.618 ********* 2021-03-09T08:49:55.594 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T08:49:55.596 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.619 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.043) 0:01:00.661 ********* 2021-03-09T08:49:55.668 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T08:49:55.669 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:55.692 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.072) 0:01:00.734 ********* 2021-03-09T08:49:55.699 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T08:49:55.701 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.725 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.033) 0:01:00.767 ********* 2021-03-09T08:49:55.774 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T08:49:55.776 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.072) 0:01:00.839 ********* 2021-03-09T08:49:55.803 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T08:49:55.805 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.828 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.030) 0:01:00.870 ********* 2021-03-09T08:49:55.866 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.890 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.061) 0:01:00.932 ********* 2021-03-09T08:49:55.907 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T08:49:55.909 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.931 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.041) 0:01:00.973 ********* 2021-03-09T08:49:55.959 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T08:49:55.961 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:55.983 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:55 +0000 (0:00:00.051) 0:01:01.025 ********* 2021-03-09T08:49:56.000 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T08:49:56.002 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.028 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.044) 0:01:01.070 ********* 2021-03-09T08:49:56.057 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T08:49:56.058 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.092 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.064) 0:01:01.134 ********* 2021-03-09T08:49:56.109 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T08:49:56.111 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:56.135 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.042) 0:01:01.177 ********* 2021-03-09T08:49:56.163 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T08:49:56.165 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:56.188 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.052) 0:01:01.230 ********* 2021-03-09T08:49:56.205 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-09T08:49:56.207 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.231 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.042) 0:01:01.272 ********* 2021-03-09T08:49:56.268 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:49:56.270 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.293 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.062) 0:01:01.335 ********* 2021-03-09T08:49:56.299 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T08:49:56.302 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.325 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.031) 0:01:01.367 ********* 2021-03-09T08:49:56.354 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T08:49:56.356 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.379 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.053) 0:01:01.421 ********* 2021-03-09T08:49:56.406 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T08:49:56.408 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:56.433 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.053) 0:01:01.475 ********* 2021-03-09T08:49:56.452 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T08:49:56.453 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.478 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.044) 0:01:01.520 ********* 2021-03-09T08:49:56.528 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T08:49:56.530 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.553 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.074) 0:01:01.594 ********* 2021-03-09T08:49:56.569 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T08:49:56.571 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.593 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.040) 0:01:01.635 ********* 2021-03-09T08:49:56.621 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T08:49:56.622 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:56.655 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.061) 0:01:01.697 ********* 2021-03-09T08:49:56.673 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T08:49:56.675 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.702 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.046) 0:01:01.743 ********* 2021-03-09T08:49:56.732 INFO:teuthology.task.ansible.out: 2021-03-09T08:49:56.734 INFO:teuthology.task.ansible.out:TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.760 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.057) 0:01:01.801 ********* 2021-03-09T08:49:56.778 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T08:49:56.779 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.803 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.043) 0:01:01.845 ********* 2021-03-09T08:49:56.831 INFO:teuthology.task.ansible.out: 2021-03-09T08:49:56.833 INFO:teuthology.task.ansible.out:TASK [common : List CDN releases available to system] ************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.859 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.055) 0:01:01.901 ********* 2021-03-09T08:49:56.877 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T08:49:56.878 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:56.905 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.045) 0:01:01.947 ********* 2021-03-09T08:49:56.923 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T08:49:56.925 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:56.952 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.047) 0:01:01.994 ********* 2021-03-09T08:49:56.981 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T08:49:56.983 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.006 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:56 +0000 (0:00:00.054) 0:01:02.048 ********* 2021-03-09T08:49:57.023 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T08:49:57.025 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.049 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.042) 0:01:02.091 ********* 2021-03-09T08:49:57.079 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T08:49:57.080 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.104 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.054) 0:01:02.146 ********* 2021-03-09T08:49:57.120 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T08:49:57.122 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:57.146 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.041) 0:01:02.187 ********* 2021-03-09T08:49:57.173 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T08:49:57.175 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.198 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.051) 0:01:02.239 ********* 2021-03-09T08:49:57.214 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:49:57.216 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.239 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.041) 0:01:02.281 ********* 2021-03-09T08:49:57.277 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T08:49:57.279 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.289 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.318 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.078) 0:01:02.360 ********* 2021-03-09T08:49:57.323 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T08:49:57.325 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.350 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.031) 0:01:02.391 ********* 2021-03-09T08:49:57.387 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T08:49:57.389 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.414 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.064) 0:01:02.456 ********* 2021-03-09T08:49:57.420 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T08:49:57.422 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.446 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.032) 0:01:02.488 ********* 2021-03-09T08:49:57.484 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:49:57.486 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.511 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.064) 0:01:02.553 ********* 2021-03-09T08:49:57.518 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T08:49:57.519 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.032) 0:01:02.586 ********* 2021-03-09T08:49:57.625 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T08:49:57.769 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.112) 0:01:02.698 ********* TASK [common : Get the current timezone] *************************************** ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.008983", "end": "2021-03-09 08:49:57.447226", "rc": 0, "start": "2021-03-09 08:49:57.438243", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.073) 0:01:02.771 ********* TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.771 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.041) 0:01:02.813 ********* 2021-03-09T08:49:57.804 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T08:49:57.806 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.834 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.062) 0:01:02.876 ********* 2021-03-09T08:49:57.862 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:49:57.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.052) 0:01:02.928 ********* 2021-03-09T08:49:57.905 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T08:49:57.907 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.930 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.043) 0:01:02.972 ********* 2021-03-09T08:49:57.960 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T08:49:57.962 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:57.989 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:57 +0000 (0:00:00.059) 0:01:03.031 ********* 2021-03-09T08:49:58.018 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:49:58.020 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.030 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.031 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.056 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.066) 0:01:03.098 ********* 2021-03-09T08:49:58.063 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T08:49:58.065 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.089 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.033) 0:01:03.131 ********* 2021-03-09T08:49:58.127 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:49:58.129 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.154 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.064) 0:01:03.196 ********* 2021-03-09T08:49:58.159 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T08:49:58.161 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.185 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.031) 0:01:03.227 ********* 2021-03-09T08:49:58.223 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T08:49:58.225 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.249 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.063) 0:01:03.291 ********* 2021-03-09T08:49:58.278 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T08:49:58.279 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.304 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.055) 0:01:03.346 ********* 2021-03-09T08:49:58.321 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T08:49:58.323 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.041) 0:01:03.387 ********* 2021-03-09T08:49:58.383 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T08:49:58.386 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.408 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.062) 0:01:03.450 ********* 2021-03-09T08:49:58.427 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T08:49:58.430 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.452 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.043) 0:01:03.494 ********* 2021-03-09T08:49:58.480 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:49:58.482 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.504 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.051) 0:01:03.546 ********* 2021-03-09T08:49:58.531 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T08:49:58.534 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.555 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.050) 0:01:03.597 ********* 2021-03-09T08:49:58.698 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:49:58.700 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.007602", "end": "2021-03-09 08:49:58.689716", "rc": 0, "start": "2021-03-09 08:49:58.682114", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T08:49:58.729 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.174) 0:01:03.771 ********* 2021-03-09T08:49:58.757 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T08:49:58.759 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.072405", "end": "2021-03-09 08:49:58.500429", "rc": 0, "start": "2021-03-09 08:49:58.428024", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:49:58.784 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.054) 0:01:03.825 ********* 2021-03-09T08:49:58.800 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T08:49:58.802 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.825 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.041) 0:01:03.867 ********* 2021-03-09T08:49:58.863 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T08:49:58.865 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.888 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.062) 0:01:03.930 ********* 2021-03-09T08:49:58.904 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T08:49:58.906 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:58.929 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:58 +0000 (0:00:00.041) 0:01:03.971 ********* 2021-03-09T08:49:59.008 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:49:59.010 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.019 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.021 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.030 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.055 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.125) 0:01:04.096 ********* 2021-03-09T08:49:59.093 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T08:49:59.094 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023718", "end": "2021-03-09 08:49:58.838761", "rc": 0, "start": "2021-03-09 08:49:58.815043", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:49:59.118 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.063) 0:01:04.160 ********* 2021-03-09T08:49:59.126 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:49:59.127 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.152 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.033) 0:01:04.194 ********* 2021-03-09T08:49:59.221 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T08:49:59.223 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.245 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.092) 0:01:04.287 ********* 2021-03-09T08:49:59.326 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T08:49:59.328 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.350 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.104) 0:01:04.392 ********* 2021-03-09T08:49:59.420 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T08:49:59.422 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.443 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.092) 0:01:04.484 ********* 2021-03-09T08:49:59.522 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T08:49:59.524 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:49:59.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:49:59 +0000 (0:00:00.101) 0:01:04.586 ********* 2021-03-09T08:50:00.507 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T08:50:00.510 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.825661", "end": "2021-03-09 08:50:00.500345", "rc": 0, "start": "2021-03-09 08:49:59.674684", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:50:00.532 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:00 +0000 (0:00:00.987) 0:01:05.573 ********* 2021-03-09T08:50:00.603 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T08:50:00.605 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:00.626 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:00 +0000 (0:00:00.094) 0:01:05.668 ********* 2021-03-09T08:50:00.830 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T08:50:00.832 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023427", "end": "2021-03-09 08:50:00.814424", "rc": 0, "start": "2021-03-09 08:50:00.790997", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:50:00.854 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:00 +0000 (0:00:00.228) 0:01:05.896 ********* 2021-03-09T08:50:02.491 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T08:50:02.493 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:02.535 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:02 +0000 (0:00:01.680) 0:01:07.577 ********* 2021-03-09T08:50:03.595 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T08:50:03.597 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:03.636 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:03 +0000 (0:00:01.100) 0:01:08.678 ********* 2021-03-09T08:50:03.750 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:03.788 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:03 +0000 (0:00:00.152) 0:01:08.830 ********* 2021-03-09T08:50:03.831 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T08:50:03.834 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:03.862 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:03 +0000 (0:00:00.073) 0:01:08.903 ********* 2021-03-09T08:50:04.610 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T08:50:04.612 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T08:50:04.636 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:04 +0000 (0:00:00.774) 0:01:09.678 ********* 2021-03-09T08:50:04.898 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T08:50:04.900 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:04.937 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:04 +0000 (0:00:00.300) 0:01:09.979 ********* 2021-03-09T08:50:05.041 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-09T08:50:05.043 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:05.079 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:05 +0000 (0:00:00.141) 0:01:10.120 ********* 2021-03-09T08:50:05.499 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T08:50:05.501 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T08:50:05.528 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:05 +0000 (0:00:00.449) 0:01:10.570 ********* 2021-03-09T08:50:05.578 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T08:50:05.580 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:05.603 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:05 +0000 (0:00:00.074) 0:01:10.645 ********* 2021-03-09T08:50:06.028 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:50:06.030 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T08:50:06.334 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T08:50:06.480 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T08:50:06.481 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:06.504 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:06 +0000 (0:00:00.901) 0:01:11.546 ********* 2021-03-09T08:50:06.626 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:50:06.628 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T08:50:06.659 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:06 +0000 (0:00:00.155) 0:01:11.701 ********* 2021-03-09T08:50:06.812 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T08:50:06.815 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:50:06.838 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:06 +0000 (0:00:00.179) 0:01:11.880 ********* 2021-03-09T08:50:06.941 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:50:06.943 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T08:50:07.215 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:50:07.216 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T08:50:07.237 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:50:07.239 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T08:50:07.500 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:50:07.502 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T08:50:07.533 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:50:07.535 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T08:50:07.560 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.721) 0:01:12.602 ********* 2021-03-09T08:50:07.714 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T08:50:07.716 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:50:07.738 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.177) 0:01:12.779 ********* 2021-03-09T08:50:07.786 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:50:07.788 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T08:50:07.815 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.077) 0:01:12.857 ********* 2021-03-09T08:50:07.907 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T08:50:07.909 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:50:07.931 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.115) 0:01:12.973 ********* 2021-03-09T08:50:08.000 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T08:50:08.002 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:08.027 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.095) 0:01:13.068 ********* 2021-03-09T08:50:08.107 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T08:50:08.109 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:50:08.136 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.109) 0:01:13.178 ********* 2021-03-09T08:50:08.146 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:50:08.146 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T08:50:08.222 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T08:50:08.224 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:08.234 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:08.245 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:08.270 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.133) 0:01:13.312 ********* 2021-03-09T08:50:08.341 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T08:50:08.343 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:08.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.094) 0:01:13.407 ********* 2021-03-09T08:50:08.457 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:50:08.458 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T08:50:08.755 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T08:50:08.780 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.415) 0:01:13.822 ********* 2021-03-09T08:50:08.880 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T08:50:08.882 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:50:08.903 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.122) 0:01:13.945 ********* 2021-03-09T08:50:08.992 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T08:50:08.994 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:09.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.112) 0:01:14.057 ********* 2021-03-09T08:50:09.095 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T08:50:09.097 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:50:09.122 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.106) 0:01:14.164 ********* 2021-03-09T08:50:09.214 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T08:50:09.216 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:09.226 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:09.228 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:09.249 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.127) 0:01:14.291 ********* 2021-03-09T08:50:09.338 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T08:50:09.340 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:09.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.116) 0:01:14.407 ********* 2021-03-09T08:50:09.373 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T08:50:09.374 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:09.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.034) 0:01:14.442 ********* 2021-03-09T08:50:09.729 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T08:50:09.731 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T08:50:09.755 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.354) 0:01:14.796 ********* 2021-03-09T08:50:10.127 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T08:50:10.129 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T08:50:10.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.395) 0:01:15.192 ********* 2021-03-09T08:50:10.345 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:50:10.347 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T08:50:10.420 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T08:50:10.422 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:10.448 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.297) 0:01:15.490 ********* 2021-03-09T08:50:10.466 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:50:10.468 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T08:50:10.498 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.049) 0:01:15.539 ********* 2021-03-09T08:50:10.578 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T08:50:10.580 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:10.611 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.113) 0:01:15.653 ********* 2021-03-09T08:50:10.691 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T08:50:10.693 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:50:10.714 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.102) 0:01:15.756 ********* 2021-03-09T08:50:10.792 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T08:50:10.794 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:10.816 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.102) 0:01:15.858 ********* 2021-03-09T08:50:10.833 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T08:50:10.835 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T08:50:10.861 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.044) 0:01:15.903 ********* 2021-03-09T08:50:11.202 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T08:50:11.203 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T08:50:11.230 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.368) 0:01:16.271 ********* 2021-03-09T08:50:11.257 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T08:50:11.258 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T08:50:11.285 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.054) 0:01:16.326 ********* 2021-03-09T08:50:11.491 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:50:11.493 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T08:50:11.629 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T08:50:11.653 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.368) 0:01:16.694 ********* 2021-03-09T08:50:11.722 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T08:50:11.724 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:11.746 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.092) 0:01:16.787 ********* 2021-03-09T08:50:11.815 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:50:11.840 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.093) 0:01:16.881 ********* 2021-03-09T08:50:11.932 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T08:50:11.934 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ActiveEnterTimestampMonotonic": "17148367", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount systemd-tmpfiles-setup.service nss-lookup.target basic.target time-sync.target sysinit.target network.target -.mount remote-fs.target system.slice local-fs.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:11 UTC", "AssertTimestampMonotonic": "17147624", "Before": "getty@tty1.service multi-user.target xdm.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ConditionTimestampMonotonic": "17147624", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-09T08:50:11.936 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1606", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ExecMainStartTimestampMonotonic": "17148341", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:11 UTC", "InactiveExitTimestampMonotonic": "17148367", "InvocationID": "9a573559ea70448abfbbaa3460818c62", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-09T08:50:11.938 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1606", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-09T08:50:11.939 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:11 UTC", "StateChangeTimestampMonotonic": "17148367", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:48:11 UTC", "WatchdogTimestampMonotonic": "17148366", "WatchdogUSec": "0"}} TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:11.960 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.120) 0:01:17.002 ********* 2021-03-09T08:50:11.982 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.021) 0:01:17.024 ********* 2021-03-09T08:50:12.052 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-09T08:50:12.054 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.075 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.093) 0:01:17.117 ********* 2021-03-09T08:50:12.144 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-09T08:50:12.146 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.174 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.098) 0:01:17.216 ********* 2021-03-09T08:50:12.256 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T08:50:12.257 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.280 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.106) 0:01:17.322 ********* 2021-03-09T08:50:12.351 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T08:50:12.353 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T08:50:12.381 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.101) 0:01:17.423 ********* 2021-03-09T08:50:12.389 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-09T08:50:12.390 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.419 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.037) 0:01:17.461 ********* 2021-03-09T08:50:12.500 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T08:50:12.502 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.524 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.104) 0:01:17.565 ********* 2021-03-09T08:50:12.593 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T08:50:12.594 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.617 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.093) 0:01:17.659 ********* 2021-03-09T08:50:12.656 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T08:50:12.659 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:05 UTC", "ActiveEnterTimestampMonotonic": "17744651", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount nss-lookup.target systemd-tmpfiles-setup.service var-run.mount network.target remote-fs.target system.slice time-sync.target sysinit.target local-fs.target systemd-journald.socket basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:05 UTC", "AssertTimestampMonotonic": "17743315", "Before": "getty@tty1.service multi-user.target xdm.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:05 UTC", "ConditionTimestampMonotonic": "17743313", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-09T08:50:12.660 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1318", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:05 UTC", "ExecMainStartTimestampMonotonic": "17744603", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:05 UTC", "InactiveExitTimestampMonotonic": "17744651", "InvocationID": "9e5f385493594ac5992e8ff4bc2436d8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-09T08:50:12.662 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1318", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-09T08:50:12.663 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:05 UTC", "StateChangeTimestampMonotonic": "17744651", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:48:05 UTC", "WatchdogTimestampMonotonic": "17744647", "WatchdogUSec": "0"}} 2021-03-09T08:50:12.687 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.069) 0:01:17.729 ********* 2021-03-09T08:50:12.704 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T08:50:12.706 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.730 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.042) 0:01:17.771 ********* 2021-03-09T08:50:12.757 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-09T08:50:12.759 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.785 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.055) 0:01:17.827 ********* 2021-03-09T08:50:12.804 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T08:50:12.806 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.835 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.049) 0:01:17.877 ********* 2021-03-09T08:50:12.875 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-09T08:50:12.877 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.899 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.063) 0:01:17.941 ********* 2021-03-09T08:50:12.926 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T08:50:12.928 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.950 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.051) 0:01:17.992 ********* 2021-03-09T08:50:12.988 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T08:50:12.990 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.018 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.067) 0:01:18.060 ********* 2021-03-09T08:50:13.036 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T08:50:13.038 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.062 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.043) 0:01:18.104 ********* 2021-03-09T08:50:13.110 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-09T08:50:13.111 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.135 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.073) 0:01:18.177 ********* 2021-03-09T08:50:13.144 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T08:50:13.145 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:50:13.171 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.035) 0:01:18.213 ********* 2021-03-09T08:50:13.202 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T08:50:13.204 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.231 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.059) 0:01:18.273 ********* 2021-03-09T08:50:13.259 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T08:50:13.261 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T08:50:13.286 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.055) 0:01:18.328 ********* 2021-03-09T08:50:13.304 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T08:50:13.306 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.329 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.042) 0:01:18.371 ********* 2021-03-09T08:50:13.371 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T08:50:13.373 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-09T08:50:13.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.070) 0:01:18.442 ********* 2021-03-09T08:50:13.537 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T08:50:13.539 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.564 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.163) 0:01:18.605 ********* 2021-03-09T08:50:13.624 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T08:50:13.626 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:50:13.651 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.087) 0:01:18.693 ********* 2021-03-09T08:50:13.777 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T08:50:13.779 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.801 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.150) 0:01:18.843 ********* 2021-03-09T08:50:13.808 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T08:50:13.810 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:50:13.835 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.033) 0:01:18.876 ********* 2021-03-09T08:50:13.999 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T08:50:14.001 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:14.033 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.198) 0:01:19.075 ********* 2021-03-09T08:50:14.040 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T08:50:14.042 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T08:50:14.068 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.034) 0:01:19.109 ********* 2021-03-09T08:50:14.095 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T08:50:14.097 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:14.121 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.053) 0:01:19.163 ********* 2021-03-09T08:50:14.139 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-09T08:50:14.140 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:14.176 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.055) 0:01:19.218 ********* 2021-03-09T08:50:14.196 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T08:50:14.198 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:50:14.223 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.047) 0:01:19.265 ********* 2021-03-09T08:50:14.251 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T08:50:14.253 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:14.279 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.055) 0:01:19.321 ********* 2021-03-09T08:50:14.319 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T08:50:14.321 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T08:50:14.357 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.077) 0:01:19.398 ********* 2021-03-09T08:50:14.514 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T08:50:14.516 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T08:50:14.539 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.182) 0:01:19.581 ********* 2021-03-09T08:50:14.609 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T08:50:14.612 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-09T08:50:14.634 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.095) 0:01:19.676 ********* 2021-03-09T08:50:14.636 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:14.864 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T08:50:14.865 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:50:14.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:14 +0000 (0:00:00.252) 0:01:19.929 ********* 2021-03-09T08:50:14.889 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:15.042 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T08:50:15.044 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:50:15.067 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.179) 0:01:20.109 ********* 2021-03-09T08:50:15.178 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T08:50:15.180 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T08:50:15.209 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.142) 0:01:20.251 ********* 2021-03-09T08:50:15.385 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T08:50:15.387 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T08:50:15.508 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.298) 0:01:20.550 ********* 2021-03-09T08:50:15.514 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T08:50:15.516 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi041"}, "changed": false} 2021-03-09T08:50:15.540 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.031) 0:01:20.582 ********* 2021-03-09T08:50:15.578 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-09T08:50:15.580 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:15.608 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.068) 0:01:20.650 ********* 2021-03-09T08:50:15.679 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T08:50:15.681 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:15.714 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.105) 0:01:20.755 ********* 2021-03-09T08:50:15.934 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T08:50:15.936 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T08:50:15.959 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:15 +0000 (0:00:00.245) 0:01:21.001 ********* 2021-03-09T08:50:16.449 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T08:50:16.450 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T08:50:16.474 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:16 +0000 (0:00:00.514) 0:01:21.515 ********* 2021-03-09T08:50:16.482 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T08:50:16.484 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi041", "ansible_hostname": "smithi041", "ansible_nodename": "smithi041"}, "changed": false, "name": "smithi041"} 2021-03-09T08:50:16.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:16 +0000 (0:00:00.043) 0:01:21.559 ********* 2021-03-09T08:50:16.570 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T08:50:16.571 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi038"}, "changed": false} 2021-03-09T08:50:16.596 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:16 +0000 (0:00:00.078) 0:01:21.637 ********* 2021-03-09T08:50:16.911 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-09T08:50:16.913 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T08:50:16.935 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:16 +0000 (0:00:00.338) 0:01:21.976 ********* 2021-03-09T08:50:17.108 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T08:50:17.110 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi038", "ansible_hostname": "smithi038", "ansible_nodename": "smithi038"}, "changed": false, "name": "smithi038"} 2021-03-09T08:50:17.135 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:17 +0000 (0:00:00.200) 0:01:22.176 ********* 2021-03-09T08:50:17.341 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T08:50:17.343 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T08:50:17.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:17 +0000 (0:00:00.231) 0:01:22.409 ********* 2021-03-09T08:50:17.532 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:17.534 INFO:teuthology.task.ansible.out:TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T08:50:17.558 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:17 +0000 (0:00:00.191) 0:01:22.600 ********* 2021-03-09T08:50:17.921 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T08:50:17.923 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T08:50:17.946 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:17 +0000 (0:00:00.387) 0:01:22.988 ********* 2021-03-09T08:50:17.964 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T08:50:17.966 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T08:50:17.991 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:17 +0000 (0:00:00.045) 0:01:23.033 ********* 2021-03-09T08:50:18.019 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T08:50:18.020 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.044 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.052) 0:01:23.086 ********* 2021-03-09T08:50:18.124 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:50:18.126 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.147 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.103) 0:01:23.189 ********* 2021-03-09T08:50:18.226 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:50:18.228 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.249 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.102) 0:01:23.291 ********* 2021-03-09T08:50:18.328 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:50:18.330 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.352 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.102) 0:01:23.394 ********* 2021-03-09T08:50:18.434 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:50:18.436 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.458 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.105) 0:01:23.500 ********* 2021-03-09T08:50:18.465 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T08:50:18.467 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T08:50:18.495 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.036) 0:01:23.537 ********* 2021-03-09T08:50:18.534 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:50:18.536 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.561 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.066) 0:01:23.603 ********* 2021-03-09T08:50:18.569 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T08:50:18.571 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.598 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.036) 0:01:23.640 ********* 2021-03-09T08:50:18.626 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:50:18.628 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.652 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.053) 0:01:23.693 ********* 2021-03-09T08:50:18.679 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:50:18.686 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.705 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.053) 0:01:23.747 ********* 2021-03-09T08:50:18.734 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T08:50:18.736 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.760 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.055) 0:01:23.802 ********* 2021-03-09T08:50:18.778 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:50:18.779 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.806 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.045) 0:01:23.848 ********* 2021-03-09T08:50:18.844 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T08:50:18.874 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.068) 0:01:23.916 ********* 2021-03-09T08:50:18.881 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:50:18.882 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:18.909 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.034) 0:01:23.950 ********* 2021-03-09T08:50:18.948 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T08:50:18.973 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:18 +0000 (0:00:00.064) 0:01:24.015 ********* 2021-03-09T08:50:18.991 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:50:18.993 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.042) 0:01:24.058 ********* 2021-03-09T08:50:19.057 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T08:50:19.081 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.064) 0:01:24.123 ********* 2021-03-09T08:50:19.099 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:50:19.101 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.134 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.053) 0:01:24.176 ********* 2021-03-09T08:50:19.174 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T08:50:19.176 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.202 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.067) 0:01:24.244 ********* 2021-03-09T08:50:19.220 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:50:19.222 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.245 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.043) 0:01:24.287 ********* 2021-03-09T08:50:19.283 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T08:50:19.285 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.310 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.064) 0:01:24.352 ********* 2021-03-09T08:50:19.327 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T08:50:19.329 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.354 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.044) 0:01:24.396 ********* 2021-03-09T08:50:19.382 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-09T08:50:19.384 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.407 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.052) 0:01:24.449 ********* 2021-03-09T08:50:19.435 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T08:50:19.459 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.052) 0:01:24.501 ********* 2021-03-09T08:50:19.487 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:50:19.489 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.516 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.056) 0:01:24.558 ********* 2021-03-09T08:50:19.533 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T08:50:19.573 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.056) 0:01:24.614 ********* 2021-03-09T08:50:19.597 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T08:50:19.599 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.630 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.057) 0:01:24.672 ********* 2021-03-09T08:50:19.650 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T08:50:19.679 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.048) 0:01:24.720 ********* 2021-03-09T08:50:19.706 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T08:50:19.708 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.735 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.056) 0:01:24.777 ********* 2021-03-09T08:50:19.753 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T08:50:19.754 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.781 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.046) 0:01:24.823 ********* 2021-03-09T08:50:19.811 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-09T08:50:19.813 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.837 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.055) 0:01:24.879 ********* 2021-03-09T08:50:19.854 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T08:50:19.856 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.888 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.050) 0:01:24.929 ********* 2021-03-09T08:50:19.916 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-09T08:50:19.917 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.941 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.053) 0:01:24.983 ********* 2021-03-09T08:50:19.957 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-09T08:50:19.959 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:19.983 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:19 +0000 (0:00:00.041) 0:01:25.025 ********* 2021-03-09T08:50:20.012 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:50:20.014 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.037 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.054) 0:01:25.079 ********* 2021-03-09T08:50:20.055 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:20.056 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.082 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.045) 0:01:25.124 ********* 2021-03-09T08:50:20.120 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:50:20.122 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.145 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.062) 0:01:25.186 ********* 2021-03-09T08:50:20.162 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T08:50:20.164 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.188 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.043) 0:01:25.230 ********* 2021-03-09T08:50:20.216 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:50:20.218 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.244 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.055) 0:01:25.285 ********* 2021-03-09T08:50:20.281 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T08:50:20.283 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.305 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.061) 0:01:25.347 ********* 2021-03-09T08:50:20.322 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T08:50:20.324 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.348 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.042) 0:01:25.390 ********* 2021-03-09T08:50:20.376 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-09T08:50:20.378 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.412 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.063) 0:01:25.454 ********* 2021-03-09T08:50:20.430 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:50:20.432 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.043) 0:01:25.497 ********* 2021-03-09T08:50:20.483 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-09T08:50:20.485 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.510 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.054) 0:01:25.551 ********* 2021-03-09T08:50:20.528 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:50:20.530 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.553 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.043) 0:01:25.595 ********* 2021-03-09T08:50:20.581 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:50:20.582 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.606 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.053) 0:01:25.648 ********* 2021-03-09T08:50:20.623 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T08:50:20.625 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.649 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.042) 0:01:25.691 ********* 2021-03-09T08:50:20.676 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:50:20.678 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.701 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.052) 0:01:25.743 ********* 2021-03-09T08:50:20.718 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T08:50:20.720 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.743 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.041) 0:01:25.785 ********* 2021-03-09T08:50:20.771 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.796 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.052) 0:01:25.838 ********* 2021-03-09T08:50:20.823 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.847 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.051) 0:01:25.889 ********* 2021-03-09T08:50:20.865 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T08:50:20.866 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.891 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.043) 0:01:25.933 ********* 2021-03-09T08:50:20.918 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T08:50:20.920 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.942 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.051) 0:01:25.984 ********* 2021-03-09T08:50:20.959 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:50:20.961 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:20.988 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:20 +0000 (0:00:00.045) 0:01:26.030 ********* 2021-03-09T08:50:21.017 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T08:50:21.019 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.042 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.053) 0:01:26.084 ********* 2021-03-09T08:50:21.059 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:50:21.061 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.085 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.042) 0:01:26.127 ********* 2021-03-09T08:50:21.113 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T08:50:21.115 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.138 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.053) 0:01:26.180 ********* 2021-03-09T08:50:21.155 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T08:50:21.157 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.181 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.042) 0:01:26.223 ********* 2021-03-09T08:50:21.219 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T08:50:21.221 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.243 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.062) 0:01:26.285 ********* 2021-03-09T08:50:21.260 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T08:50:21.262 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.285 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.041) 0:01:26.327 ********* 2021-03-09T08:50:21.313 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T08:50:21.315 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.340 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.054) 0:01:26.381 ********* 2021-03-09T08:50:21.356 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T08:50:21.358 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.381 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.041) 0:01:26.423 ********* 2021-03-09T08:50:21.462 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T08:50:21.463 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.486 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.104) 0:01:26.527 ********* 2021-03-09T08:50:21.524 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T08:50:21.526 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005059", "end": "2021-03-09 08:50:21.286926", "rc": 0, "start": "2021-03-09 08:50:21.281867", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:50:21.551 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.065) 0:01:26.593 ********* 2021-03-09T08:50:21.568 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T08:50:21.570 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.594 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.043) 0:01:26.636 ********* 2021-03-09T08:50:21.622 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T08:50:21.653 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.059) 0:01:26.695 ********* 2021-03-09T08:50:21.672 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T08:50:21.673 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.698 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.044) 0:01:26.740 ********* 2021-03-09T08:50:21.769 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T08:50:21.771 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.098) 0:01:26.839 ********* 2021-03-09T08:50:21.866 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T08:50:21.868 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:21.892 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:21 +0000 (0:00:00.094) 0:01:26.934 ********* 2021-03-09T08:50:22.014 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T08:50:22.016 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T08:50:22.039 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:22 +0000 (0:00:00.147) 0:01:27.081 ********* 2021-03-09T08:50:22.097 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T08:50:22.099 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003580", "end": "2021-03-09 08:50:22.074641", "rc": 0, "start": "2021-03-09 08:50:22.071061", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:50:22.122 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:22 +0000 (0:00:00.083) 0:01:27.164 ********* 2021-03-09T08:50:22.128 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T08:50:22.130 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:22.153 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:22 +0000 (0:00:00.031) 0:01:27.195 ********* 2021-03-09T08:50:22.192 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T08:50:22.216 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:22 +0000 (0:00:00.062) 0:01:27.258 ********* 2021-03-09T08:50:22.601 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T08:50:22.602 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T08:50:22.626 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:22 +0000 (0:00:00.409) 0:01:27.668 ********* 2021-03-09T08:50:22.695 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T08:50:22.697 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:22.720 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:22 +0000 (0:00:00.094) 0:01:27.762 ********* 2021-03-09T08:50:23.521 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T08:50:23.523 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T08:50:23.766 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T08:50:24.478 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T08:50:24.502 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:24 +0000 (0:00:01.781) 0:01:29.544 ********* 2021-03-09T08:50:24.749 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T08:50:24.774 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:24 +0000 (0:00:00.271) 0:01:29.816 ********* 2021-03-09T08:50:25.497 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T08:50:25.499 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:25.521 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:25 +0000 (0:00:00.747) 0:01:30.563 ********* 2021-03-09T08:50:25.590 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T08:50:25.614 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:25 +0000 (0:00:00.092) 0:01:30.655 ********* 2021-03-09T08:50:25.796 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T08:50:25.798 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:25.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:25 +0000 (0:00:00.206) 0:01:30.862 ********* 2021-03-09T08:50:25.838 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T08:50:25.840 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036514", "end": "2021-03-09 08:50:25.598014", "rc": 0, "start": "2021-03-09 08:50:25.561500", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:50:25.865 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:25 +0000 (0:00:00.044) 0:01:30.907 ********* 2021-03-09T08:50:25.892 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T08:50:25.919 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:25 +0000 (0:00:00.053) 0:01:30.961 ********* 2021-03-09T08:50:26.155 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T08:50:26.156 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036861", "end": "2021-03-09 08:50:26.125811", "rc": 0, "start": "2021-03-09 08:50:26.088950", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:50:26.179 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:26 +0000 (0:00:00.259) 0:01:31.221 ********* 2021-03-09T08:50:28.832 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T08:50:28.834 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:28.858 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:28 +0000 (0:00:02.679) 0:01:33.900 ********* 2021-03-09T08:50:29.460 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:29.487 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:29 +0000 (0:00:00.629) 0:01:34.529 ********* 2021-03-09T08:50:39.069 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T08:50:39.070 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-09T08:50:39.072 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-09T08:50:39.074 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T08:50:39.098 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:39 +0000 (0:00:09.610) 0:01:44.140 ********* 2021-03-09T08:50:39.432 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-09T08:50:39.434 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-09T08:50:39.436 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T08:50:39.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:39 +0000 (0:00:00.357) 0:01:44.498 ********* 2021-03-09T08:50:40.075 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:50:40.077 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:40.102 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:40 +0000 (0:00:00.645) 0:01:45.144 ********* 2021-03-09T08:50:40.183 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T08:50:40.185 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:40.206 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:40 +0000 (0:00:00.104) 0:01:45.248 ********* 2021-03-09T08:50:40.401 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:50:40.403 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:40.428 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:40 +0000 (0:00:00.221) 0:01:45.469 ********* 2021-03-09T08:50:40.507 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T08:50:40.509 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:40.532 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:40 +0000 (0:00:00.104) 0:01:45.574 ********* 2021-03-09T08:50:49.244 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:50:49.246 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9858 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-03-09T08:50:49.248 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-03-09T08:50:49.249 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9858 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-03-09T08:50:49.251 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T08:50:49.272 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:49 +0000 (0:00:08.739) 0:01:54.314 ********* 2021-03-09T08:50:49.312 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9718 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-03-09T08:50:49.314 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-03-09T08:50:49.315 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9718 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-03-09T08:50:49.317 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T08:50:49.341 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:49 +0000 (0:00:00.068) 0:01:54.382 ********* 2021-03-09T08:50:50.233 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-09T08:50:50.235 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:50.259 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:50 +0000 (0:00:00.918) 0:01:55.301 ********* 2021-03-09T08:50:50.309 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:50.333 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:50 +0000 (0:00:00.074) 0:01:55.375 ********* 2021-03-09T08:50:51.184 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T08:50:51.186 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:50:51.209 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.875) 0:01:56.251 ********* 2021-03-09T08:50:51.289 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:50:51.313 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.104) 0:01:56.355 ********* 2021-03-09T08:50:51.393 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T08:50:51.395 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T08:50:51.424 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.110) 0:01:56.466 ********* 2021-03-09T08:50:51.503 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T08:50:51.505 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:51.527 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.103) 0:01:56.569 ********* 2021-03-09T08:50:51.532 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T08:50:51.534 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T08:50:51.559 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.031) 0:01:56.601 ********* 2021-03-09T08:50:51.649 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T08:50:51.652 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:51.678 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.119) 0:01:56.720 ********* 2021-03-09T08:50:51.707 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T08:50:51.709 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T08:50:51.736 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.057) 0:01:56.778 ********* 2021-03-09T08:50:51.876 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T08:50:52.043 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:51 +0000 (0:00:00.165) 0:01:56.943 ********* 2021-03-09T08:50:52.276 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T08:50:52.278 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T08:50:52.301 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:52 +0000 (0:00:00.399) 0:01:57.343 ********* 2021-03-09T08:50:52.319 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T08:50:52.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:52 +0000 (0:00:00.042) 0:01:57.385 ********* 2021-03-09T08:50:52.516 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T08:50:52.518 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T08:50:52.540 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:52 +0000 (0:00:00.196) 0:01:57.582 ********* 2021-03-09T08:50:52.579 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T08:50:52.604 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:52 +0000 (0:00:00.063) 0:01:57.646 ********* 2021-03-09T08:50:52.737 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T08:50:52.739 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:52.762 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:52 +0000 (0:00:00.158) 0:01:57.804 ********* 2021-03-09T08:50:52.812 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:52.841 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:52 +0000 (0:00:00.078) 0:01:57.883 ********* 2021-03-09T08:50:53.110 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T08:50:53.111 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:12 UTC", "ActiveEnterTimestampMonotonic": "17567111", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target -.mount network.target systemd-tmpfiles-setup.service remote-fs.target sysinit.target system.slice nss-lookup.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:11 UTC", "AssertTimestampMonotonic": "17146872", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ConditionTimestampMonotonic": "17146872", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-09T08:50:53.113 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1715", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:12 UTC", "ExecMainStartTimestampMonotonic": "17567099", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:11 UTC", "InactiveExitTimestampMonotonic": "17147570", "InvocationID": "4ad68389025c42a39c80906dc8b57fba", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-09T08:50:53.115 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1715", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-09T08:50:53.116 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:12 UTC", "StateChangeTimestampMonotonic": "17567111", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:48:12 UTC", "WatchdogTimestampMonotonic": "17567109", "WatchdogUSec": "0"}} 2021-03-09T08:50:53.136 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.295) 0:01:58.178 ********* 2021-03-09T08:50:53.206 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:06 UTC", "ActiveEnterTimestampMonotonic": "18326731", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target -.mount systemd-tmpfiles-setup.service nss-lookup.target remote-fs.target system.slice systemd-journald.socket network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:05 UTC", "AssertTimestampMonotonic": "17744826", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:05 UTC", "ConditionTimestampMonotonic": "17744825", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-09T08:50:53.208 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1423", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:06 UTC", "ExecMainStartTimestampMonotonic": "18326719", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:05 UTC", "InactiveExitTimestampMonotonic": "17745933", "InvocationID": "4c612d8949cf493a9d390e0bf38a691a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-09T08:50:53.210 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1423", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-09T08:50:53.211 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:06 UTC", "StateChangeTimestampMonotonic": "18326731", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:48:06 UTC", "WatchdogTimestampMonotonic": "18326730", "WatchdogUSec": "0"}} TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.231 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.094) 0:01:58.273 ********* 2021-03-09T08:50:53.266 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.034) 0:01:58.308 ********* 2021-03-09T08:50:53.299 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.325 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.058) 0:01:58.366 ********* 2021-03-09T08:50:53.341 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:50:53.342 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.042) 0:01:58.409 ********* 2021-03-09T08:50:53.395 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.425 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.057) 0:01:58.467 ********* 2021-03-09T08:50:53.455 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:50:53.457 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.481 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.055) 0:01:58.523 ********* 2021-03-09T08:50:53.508 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.532 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.051) 0:01:58.574 ********* 2021-03-09T08:50:53.550 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:53.552 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.043) 0:01:58.617 ********* 2021-03-09T08:50:53.603 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.641 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.064) 0:01:58.682 ********* 2021-03-09T08:50:53.658 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:50:53.660 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.683 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.043) 0:01:58.725 ********* 2021-03-09T08:50:53.710 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.743 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.059) 0:01:58.784 ********* 2021-03-09T08:50:53.760 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:50:53.762 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.789 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.046) 0:01:58.831 ********* 2021-03-09T08:50:53.816 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.052) 0:01:58.883 ********* 2021-03-09T08:50:53.860 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:50:53.862 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.045) 0:01:58.929 ********* 2021-03-09T08:50:53.916 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.944 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.057) 0:01:58.986 ********* 2021-03-09T08:50:53.962 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:50:53.964 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:53.993 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:53 +0000 (0:00:00.048) 0:01:59.034 ********* 2021-03-09T08:50:54.020 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.044 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.052) 0:01:59.086 ********* 2021-03-09T08:50:54.073 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.098 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.053) 0:01:59.139 ********* 2021-03-09T08:50:54.116 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.142 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.044) 0:01:59.184 ********* 2021-03-09T08:50:54.170 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:50:54.172 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.196 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.053) 0:01:59.237 ********* 2021-03-09T08:50:54.223 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.251 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.055) 0:01:59.293 ********* 2021-03-09T08:50:54.269 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:50:54.270 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.047) 0:01:59.341 ********* 2021-03-09T08:50:54.319 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.046) 0:01:59.388 ********* 2021-03-09T08:50:54.365 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:50:54.366 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.392 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.045) 0:01:59.433 ********* 2021-03-09T08:50:54.409 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.434 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.042) 0:01:59.476 ********* 2021-03-09T08:50:54.462 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:50:54.464 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.488 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.053) 0:01:59.530 ********* 2021-03-09T08:50:54.506 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.531 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.043) 0:01:59.573 ********* 2021-03-09T08:50:54.558 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:50:54.560 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.584 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.052) 0:01:59.626 ********* 2021-03-09T08:50:54.611 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.636 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.052) 0:01:59.678 ********* 2021-03-09T08:50:54.655 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:50:54.657 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.681 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.044) 0:01:59.723 ********* 2021-03-09T08:50:54.710 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.734 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.052) 0:01:59.775 ********* 2021-03-09T08:50:54.761 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:50:54.763 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.786 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.051) 0:01:59.827 ********* 2021-03-09T08:50:54.802 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.831 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.045) 0:01:59.873 ********* 2021-03-09T08:50:54.859 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:50:54.860 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.884 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.053) 0:01:59.926 ********* 2021-03-09T08:50:54.912 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.938 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.052) 0:01:59.979 ********* 2021-03-09T08:50:54.955 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:50:54.957 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.983 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:00.045) 0:02:00.024 ********* 2021-03-09T08:50:55.000 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.026 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.043) 0:02:00.068 ********* 2021-03-09T08:50:55.053 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:50:55.055 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.079 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.052) 0:02:00.121 ********* 2021-03-09T08:50:55.096 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.122 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.042) 0:02:00.163 ********* 2021-03-09T08:50:55.149 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:50:55.151 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.175 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.053) 0:02:00.217 ********* 2021-03-09T08:50:55.202 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.228 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.053) 0:02:00.270 ********* 2021-03-09T08:50:55.246 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:50:55.248 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.273 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.045) 0:02:00.315 ********* 2021-03-09T08:50:55.301 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.325 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.052) 0:02:00.367 ********* 2021-03-09T08:50:55.344 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:50:55.346 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.369 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.043) 0:02:00.411 ********* 2021-03-09T08:50:55.399 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.427 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.057) 0:02:00.469 ********* 2021-03-09T08:50:55.456 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:50:55.458 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.480 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.053) 0:02:00.522 ********* 2021-03-09T08:50:55.497 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.523 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.042) 0:02:00.564 ********* 2021-03-09T08:50:55.550 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:50:55.552 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.053) 0:02:00.618 ********* 2021-03-09T08:50:55.593 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.620 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.043) 0:02:00.661 ********* 2021-03-09T08:50:55.648 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:50:55.650 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.675 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.055) 0:02:00.717 ********* 2021-03-09T08:50:55.693 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.718 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.042) 0:02:00.760 ********* 2021-03-09T08:50:55.746 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T08:50:55.748 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.773 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.054) 0:02:00.815 ********* 2021-03-09T08:50:55.790 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.819 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.045) 0:02:00.860 ********* 2021-03-09T08:50:55.964 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:55.966 INFO:teuthology.task.ansible.out:TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T08:50:55.992 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:55 +0000 (0:00:00.173) 0:02:01.034 ********* 2021-03-09T08:50:55.994 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:56.023 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T08:50:56.049 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.056) 0:02:01.091 ********* 2021-03-09T08:50:56.382 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:50:56.384 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T08:50:56.413 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.363) 0:02:01.454 ********* 2021-03-09T08:50:56.432 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T08:50:56.459 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.046) 0:02:01.501 ********* 2021-03-09T08:50:56.486 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:50:56.488 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.513 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.053) 0:02:01.555 ********* 2021-03-09T08:50:56.530 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.555 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.042) 0:02:01.597 ********* 2021-03-09T08:50:56.557 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:56.898 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T08:50:56.900 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T08:50:56.922 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.366) 0:02:01.963 ********* 2021-03-09T08:50:56.973 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T08:50:56.999 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.077) 0:02:02.041 ********* 2021-03-09T08:50:57.360 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T08:50:57.401 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:57 +0000 (0:00:00.402) 0:02:02.443 ********* 2021-03-09T08:50:57.423 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T08:50:57.447 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:57 +0000 (0:00:00.045) 0:02:02.488 ********* 2021-03-09T08:50:57.856 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T08:50:57.858 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T08:50:57.889 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:57 +0000 (0:00:00.442) 0:02:02.931 ********* 2021-03-09T08:50:57.900 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T08:50:57.937 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:57 +0000 (0:00:00.048) 0:02:02.979 ********* 2021-03-09T08:50:58.127 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:58.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.228) 0:02:03.208 ********* 2021-03-09T08:50:58.175 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:58.216 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.050) 0:02:03.258 ********* 2021-03-09T08:50:58.247 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T08:50:58.250 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.273 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.057) 0:02:03.315 ********* 2021-03-09T08:50:58.303 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.330 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.055) 0:02:03.371 ********* 2021-03-09T08:50:58.348 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-09T08:50:58.350 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.044) 0:02:03.416 ********* 2021-03-09T08:50:58.403 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.429 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.054) 0:02:03.470 ********* 2021-03-09T08:50:58.479 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T08:50:58.482 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.504 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.075) 0:02:03.546 ********* 2021-03-09T08:50:58.512 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.536 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.032) 0:02:03.578 ********* 2021-03-09T08:50:58.584 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T08:50:58.587 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.611 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.074) 0:02:03.652 ********* 2021-03-09T08:50:58.617 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.643 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.031) 0:02:03.684 ********* 2021-03-09T08:50:58.697 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T08:50:58.699 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.722 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.079) 0:02:03.764 ********* 2021-03-09T08:50:58.730 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.761 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.039) 0:02:03.803 ********* 2021-03-09T08:50:58.801 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:50:58.803 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T08:50:58.826 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.064) 0:02:03.868 ********* 2021-03-09T08:50:58.856 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T08:50:58.881 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:58 +0000 (0:00:00.054) 0:02:03.922 ********* 2021-03-09T08:50:58.918 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T08:50:58.920 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.930 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.932 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.941 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.943 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.955 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T08:50:58.967 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.969 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.969 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.981 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.983 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.984 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:58.996 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.007 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.009 INFO:teuthology.task.ansible.out: skipping: [smithi038.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.010 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi041.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.044 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.163) 0:02:04.086 ********* 2021-03-09T08:50:59.051 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.052 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.053 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.054 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.065 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.099 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.054) 0:02:04.140 ********* 2021-03-09T08:50:59.368 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003664", "end": "2021-03-09 08:50:59.340214", "rc": 0, "start": "2021-03-09 08:50:59.336550", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004056", "end": "2021-03-09 08:50:59.342767", "rc": 0, "start": "2021-03-09 08:50:59.338711", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T08:50:59.396 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.297) 0:02:04.438 ********* 2021-03-09T08:50:59.427 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.030) 0:02:04.468 ********* 2021-03-09T08:50:59.478 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T08:50:59.480 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.508 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.081) 0:02:04.550 ********* 2021-03-09T08:50:59.515 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.548 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.039) 0:02:04.590 ********* 2021-03-09T08:50:59.618 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-09T08:50:59.620 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.650 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.101) 0:02:04.691 ********* 2021-03-09T08:50:59.660 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.686 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.036) 0:02:04.728 ********* 2021-03-09T08:50:59.736 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T08:50:59.738 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.766 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.079) 0:02:04.808 ********* 2021-03-09T08:50:59.774 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:59.802 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:59 +0000 (0:00:00.035) 0:02:04.844 ********* 2021-03-09T08:51:00.775 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T08:51:00.777 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:51:00.812 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:00 +0000 (0:00:01.010) 0:02:05.854 ********* 2021-03-09T08:51:00.835 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:51:00.864 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:00 +0000 (0:00:00.051) 0:02:05.906 ********* 2021-03-09T08:51:00.907 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T08:51:00.908 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:00.934 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:00 +0000 (0:00:00.070) 0:02:05.976 ********* 2021-03-09T08:51:00.962 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:00.987 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:00 +0000 (0:00:00.052) 0:02:06.029 ********* 2021-03-09T08:51:01.047 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:51:01.049 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'block_used': 13643752, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451356, 'size_available': 927544754176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113942, 'block_size': 4096, 'inode_available': 60941034}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451356, "block_size": 4096, "block_total": 240095108, "block_used": 13643752, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941034, "inode_total": 61054976, "inode_used": 113942, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544754176, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-09T08:51:01.075 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.087) 0:02:06.116 ********* 2021-03-09T08:51:01.082 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'block_used': 13643752, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451356, 'size_available': 927544754176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113942, 'block_size': 4096, 'inode_available': 60941034}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451356, "block_size": 4096, "block_total": 240095108, "block_used": 13643752, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941034, "inode_total": 61054976, "inode_used": 113942, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544754176, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-09T08:51:01.109 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.034) 0:02:06.151 ********* 2021-03-09T08:51:01.285 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T08:51:01.287 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009936", "end": "2021-03-09 08:51:01.244061", "rc": 0, "start": "2021-03-09 08:51:01.234125", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:01.310 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.200) 0:02:06.352 ********* 2021-03-09T08:51:01.329 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009846", "end": "2021-03-09 08:51:01.089054", "rc": 0, "start": "2021-03-09 08:51:01.079208", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:01.359 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.049) 0:02:06.401 ********* 2021-03-09T08:51:01.398 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T08:51:01.400 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'block_used': 13643752, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451356, 'size_available': 927544754176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113942, 'block_size': 4096, 'inode_available': 60941034}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451356, "block_size": 4096, "block_total": 240095108, "block_used": 13643752, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941034, "inode_total": 61054976, "inode_used": 113942, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544754176, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:01.427 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.068) 0:02:06.469 ********* 2021-03-09T08:51:01.447 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'block_used': 13643752, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451356, 'size_available': 927544754176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113942, 'block_size': 4096, 'inode_available': 60941034}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451356, "block_size": 4096, "block_total": 240095108, "block_used": 13643752, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941034, "inode_total": 61054976, "inode_used": 113942, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544754176, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:01.477 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.049) 0:02:06.519 ********* 2021-03-09T08:51:01.494 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T08:51:01.523 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.045) 0:02:06.564 ********* 2021-03-09T08:51:01.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:01 +0000 (0:00:00.053) 0:02:06.618 ********* 2021-03-09T08:51:01.641 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:51:01.643 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:01.685 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:03.369 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN5UWwtMU4c012eVWdRRMoRa32tIJevSvZ'], 'uuids': ['ace6fbd5-6f29-42f5-811d-dfbf274652b4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.543117", "end": "2021-03-09 08:51:03.142917", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN5UWwtMU4c012eVWdRRMoRa32tIJevSvZ"], "labels": [], "masters": [], "uuids": ["ace6fbd5-6f29-42f5-811d-dfbf274652b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:01.599800", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:03.370 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:03.381 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:03.392 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:03.427 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reJJNAnjDd1gQHGbcbovRCkQ91BkUrwdWG'], 'uuids': ['2264d899-4608-4c64-b2f4-8854b121603c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.649267", "end": "2021-03-09 08:51:03.375123", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reJJNAnjDd1gQHGbcbovRCkQ91BkUrwdWG"], "labels": [], "masters": [], "uuids": ["2264d899-4608-4c64-b2f4-8854b121603c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:01.725856", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:03.439 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:03.451 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.650 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN3lJ8JN92yFthw6YN3t2CN0bVbduwjJlX'], 'uuids': ['7b5d1e35-25a3-4328-9c6b-67b4aab7bb1f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.121260", "end": "2021-03-09 08:51:04.414830", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN3lJ8JN92yFthw6YN3t2CN0bVbduwjJlX"], "labels": [], "masters": [], "uuids": ["7b5d1e35-25a3-4328-9c6b-67b4aab7bb1f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:03.293570", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:04.651 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S', 'wwn-0x5000c500871a27d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1', 'wwn-0x5000c500871a27d9-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a27d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HM0S', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S", "wwn-0x5000c500871a27d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1", "wwn-0x5000c500871a27d9-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HM0S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a27d9"}}, "skip_reason": 2021-03-09T08:51:04.653 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T08:51:04.706 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reuCRT319DZ9JrvMfk7vvA9QGXnjRxlDSj'], 'uuids': ['8a2cbe53-8581-4621-a647-022ea866b999']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.123060", "end": "2021-03-09 08:51:04.658187", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reuCRT319DZ9JrvMfk7vvA9QGXnjRxlDSj"], "labels": [], "masters": [], "uuids": ["8a2cbe53-8581-4621-a647-022ea866b999"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:03.535127", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:04.708 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D', 'wwn-0x5000c500871b5cd5'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D-part1', 'wwn-0x5000c500871b5cd5-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b5cd5', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J33D', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D", "wwn-0x5000c500871b5cd5"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D-part1", "wwn-0x5000c500871b5cd5-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J33D", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b5cd5"}}, "skip_reason": 2021-03-09T08:51:04.709 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T08:51:05.934 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-czJeQG-o4qs-46Gu-wRar-t3UW-yvJ6-y8v1DG', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN', 'nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.141325", "end": "2021-03-09 08:51:05.701762", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-czJeQG-o4qs-46Gu-wRar-t3UW-yvJ6-y8v1DG", "nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN", "nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:04.560437", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-09T08:51:05.938 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:05.947 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.980 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-UUCPSm-52Py-VeDi-3jnS-d3ik-MZZb-JlTYtm', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.140955", "end": "2021-03-09 08:51:05.930190", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-UUCPSm-52Py-VeDi-3jnS-d3ik-MZZb-JlTYtm", "nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:04.789235", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT 2021-03-09T08:51:05.981 INFO:teuthology.task.ansible.out: data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:05.991 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.003 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.246 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNAiFUStaBNs94lkCfT2myENbRG0XOt5KJ'], 'uuids': ['bc9dacbb-a736-45f4-8454-48e12dc9a711']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.161395", "end": "2021-03-09 08:51:07.013226", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNAiFUStaBNs94lkCfT2myENbRG0XOt5KJ"], "labels": [], "masters": [], "uuids": ["bc9dacbb-a736-45f4-8454-48e12dc9a711"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:05.851831", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:07.258 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.259 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.272 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reVjCxKNoC4AKvKsRz9gJMn6rEFamzB3GW'], 'uuids': ['ba8591b0-82c1-478f-8290-d62bd079fd07']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.132156", "end": "2021-03-09 08:51:07.231753", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reVjCxKNoC4AKvKsRz9gJMn6rEFamzB3GW"], "labels": [], "masters": [], "uuids": ["ba8591b0-82c1-478f-8290-d62bd079fd07"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:06.099597", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:07.283 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.295 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:08.521 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNSkXvV9dMx4feD0yqHduq149DPgItEHuN'], 'uuids': ['a7092f92-0535-43f7-bc17-233d845b40c6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.127934", "end": "2021-03-09 08:51:08.293301", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNSkXvV9dMx4feD0yqHduq149DPgItEHuN"], "labels": [], "masters": [], "uuids": ["a7092f92-0535-43f7-bc17-233d845b40c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:07.165367", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:08.533 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:08.545 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reCp7RiO7JqFq0r1EYTv7nT0bdIfxYDS7A'], 'uuids': ['941542a3-5df5-42c3-b089-24d716adea40']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.126224", "end": "2021-03-09 08:51:08.497557", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reCp7RiO7JqFq0r1EYTv7nT0bdIfxYDS7A"], "labels": [], "masters": [], "uuids": ["941542a3-5df5-42c3-b089-24d716adea40"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:07.371333", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:08.556 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:09.801 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNISUlp2gea7fSDOawZPUw50vB7TDFC1zi'], 'uuids': ['275f3bea-98fc-409d-bf0d-c27feedc2191']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.132616", "end": "2021-03-09 08:51:09.576961", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNISUlp2gea7fSDOawZPUw50vB7TDFC1zi"], "labels": [], "masters": [], "uuids": ["275f3bea-98fc-409d-bf0d-c27feedc2191"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:08.444345", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:09.803 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reD0w1Nuw2ywxfYEx8M3qYmJurZLSWof45'], 'uuids': ['9f247ed0-be26-44f2-affa-f41fb95d86af']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.125899", "end": "2021-03-09 08:51:09.755456", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reD0w1Nuw2ywxfYEx8M3qYmJurZLSWof45"], "labels": [], "masters": [], "uuids": ["9f247ed0-be26-44f2-affa-f41fb95d86af"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:08.629557", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:51:09.850 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:08.274) 0:02:14.892 ********* 2021-03-09T08:51:09.876 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:00.026) 0:02:14.918 ********* 2021-03-09T08:51:09.948 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:51:09.949 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:09.971 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.236 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reJJNAnjDd1gQHGbcbovRCkQ91BkUrwdWG'], 'uuids': ['2264d899-4608-4c64-b2f4-8854b121603c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.168309", "end": "2021-03-09 08:51:10.192345", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reJJNAnjDd1gQHGbcbovRCkQ91BkUrwdWG"], "labels": [], "masters": [], "uuids": ["2264d899-4608-4c64-b2f4-8854b121603c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.024036", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.165593 s, 697 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.165593 s, 697 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.248 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.250 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.263 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN5UWwtMU4c012eVWdRRMoRa32tIJevSvZ'], 'uuids': ['ace6fbd5-6f29-42f5-811d-dfbf274652b4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156437", "end": "2021-03-09 08:51:10.026521", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN5UWwtMU4c012eVWdRRMoRa32tIJevSvZ"], "labels": [], "masters": [], "uuids": ["ace6fbd5-6f29-42f5-811d-dfbf274652b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:09.870084", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151666 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151666 s, 761 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.266 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.268 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.552 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reuCRT319DZ9JrvMfk7vvA9QGXnjRxlDSj'], 'uuids': ['8a2cbe53-8581-4621-a647-022ea866b999']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.166008", "end": "2021-03-09 08:51:10.503843", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reuCRT319DZ9JrvMfk7vvA9QGXnjRxlDSj"], "labels": [], "masters": [], "uuids": ["8a2cbe53-8581-4621-a647-022ea866b999"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.337835", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161443 s, 714 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161443 s, 714 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.564 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN3lJ8JN92yFthw6YN3t2CN0bVbduwjJlX'], 'uuids': ['7b5d1e35-25a3-4328-9c6b-67b4aab7bb1f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.154650", "end": "2021-03-09 08:51:10.330757", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZN3lJ8JN92yFthw6YN3t2CN0bVbduwjJlX"], "labels": [], "masters": [], "uuids": ["7b5d1e35-25a3-4328-9c6b-67b4aab7bb1f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.176107", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149855 s, 770 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149855 s, 770 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.565 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D', 'wwn-0x5000c500871b5cd5'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D-part1', 'wwn-0x5000c500871b5cd5-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b5cd5', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J33D', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D", "wwn-0x5000c500871b5cd5"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D-part1", "wwn-0x5000c500871b5cd5-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J33D", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b5cd5"}}, "skip_reason": 2021-03-09T08:51:10.568 INFO:teuthology.task.ansible.out:"Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S', 'wwn-0x5000c500871a27d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1', 'wwn-0x5000c500871a27d9-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a27d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HM0S', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S", "wwn-0x5000c500871a27d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1", "wwn-0x5000c500871a27d9-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HM0S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x500 2021-03-09T08:51:10.570 INFO:teuthology.task.ansible.out:0c500871a27d9"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.735 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-UUCPSm-52Py-VeDi-3jnS-d3ik-MZZb-JlTYtm', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.042527", "end": "2021-03-09 08:51:10.684680", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-UUCPSm-52Py-VeDi-3jnS-d3ik-MZZb-JlTYtm", "nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.642153", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0397105 s, 2.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0397105 s, 2.9 GB 2021-03-09T08:51:10.737 INFO:teuthology.task.ansible.out:/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.748 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.749 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-czJeQG-o4qs-46Gu-wRar-t3UW-yvJ6-y8v1DG', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN', 'nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052906", "end": "2021-03-09 08:51:10.521653", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-czJeQG-o4qs-46Gu-wRar-t3UW-yvJ6-y8v1DG", "nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN", "nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.468747", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0480389 s, 2.4 GB/s", "std 2021-03-09T08:51:10.751 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0480389 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.761 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.762 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.764 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.048 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reVjCxKNoC4AKvKsRz9gJMn6rEFamzB3GW'], 'uuids': ['ba8591b0-82c1-478f-8290-d62bd079fd07']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.167557", "end": "2021-03-09 08:51:11.003376", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reVjCxKNoC4AKvKsRz9gJMn6rEFamzB3GW"], "labels": [], "masters": [], "uuids": ["ba8591b0-82c1-478f-8290-d62bd079fd07"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.835819", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.163859 s, 704 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.163859 s, 704 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.059 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.060 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.228 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNAiFUStaBNs94lkCfT2myENbRG0XOt5KJ'], 'uuids': ['bc9dacbb-a736-45f4-8454-48e12dc9a711']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.167075", "end": "2021-03-09 08:51:10.999104", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNAiFUStaBNs94lkCfT2myENbRG0XOt5KJ"], "labels": [], "masters": [], "uuids": ["bc9dacbb-a736-45f4-8454-48e12dc9a711"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:10.832029", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.162476 s, 710 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.162476 s, 710 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.229 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.242 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.496 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reCp7RiO7JqFq0r1EYTv7nT0bdIfxYDS7A'], 'uuids': ['941542a3-5df5-42c3-b089-24d716adea40']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.166645", "end": "2021-03-09 08:51:11.444843", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reCp7RiO7JqFq0r1EYTv7nT0bdIfxYDS7A"], "labels": [], "masters": [], "uuids": ["941542a3-5df5-42c3-b089-24d716adea40"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:11.278198", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.16343 s, 706 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.16343 s, 706 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.508 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.530 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNSkXvV9dMx4feD0yqHduq149DPgItEHuN'], 'uuids': ['a7092f92-0535-43f7-bc17-233d845b40c6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.157825", "end": "2021-03-09 08:51:11.302945", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNSkXvV9dMx4feD0yqHduq149DPgItEHuN"], "labels": [], "masters": [], "uuids": ["a7092f92-0535-43f7-bc17-233d845b40c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:11.145120", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152983 s, 754 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152983 s, 754 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.541 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.791 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reD0w1Nuw2ywxfYEx8M3qYmJurZLSWof45'], 'uuids': ['9f247ed0-be26-44f2-affa-f41fb95d86af']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.165264", "end": "2021-03-09 08:51:11.746407", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-wqKoXeM3XWeQNrB3tkAH0evC7RPYk2reD0w1Nuw2ywxfYEx8M3qYmJurZLSWof45"], "labels": [], "masters": [], "uuids": ["9f247ed0-be26-44f2-affa-f41fb95d86af"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:11.581143", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160717 s, 718 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160717 s, 718 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.827 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:11 +0000 (0:00:01.950) 0:02:16.868 ********* 2021-03-09T08:51:11.835 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNISUlp2gea7fSDOawZPUw50vB7TDFC1zi'], 'uuids': ['275f3bea-98fc-409d-bf0d-c27feedc2191']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.160896", "end": "2021-03-09 08:51:11.599210", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-iYZXfig1z8HF9TR6Mj6suhrostxFEnZNISUlp2gea7fSDOawZPUw50vB7TDFC1zi"], "labels": [], "masters": [], "uuids": ["275f3bea-98fc-409d-bf0d-c27feedc2191"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:51:11.438314", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156026 s, 739 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156026 s, 739 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.874 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:11 +0000 (0:00:00.047) 0:02:16.916 ********* 2021-03-09T08:51:12.111 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T08:51:12.113 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.096807", "end": "2021-03-09 08:51:12.021651", "rc": 0, "start": "2021-03-09 08:51:11.924844", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:12.138 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.263) 0:02:17.179 ********* 2021-03-09T08:51:12.187 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.118034", "end": "2021-03-09 08:51:11.917630", "rc": 0, "start": "2021-03-09 08:51:11.799596", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:12.214 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.076) 0:02:17.256 ********* 2021-03-09T08:51:12.358 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-09T08:51:12.360 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.076134", "end": "2021-03-09 08:51:12.310985", "rc": 0, "start": "2021-03-09 08:51:12.234851", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:12.386 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.171) 0:02:17.428 ********* 2021-03-09T08:51:12.405 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.054886", "end": "2021-03-09 08:51:12.191043", "rc": 0, "start": "2021-03-09 08:51:12.136157", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:12.433 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.046) 0:02:17.475 ********* 2021-03-09T08:51:12.461 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T08:51:12.523 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.089) 0:02:17.565 ********* 2021-03-09T08:51:12.560 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.036) 0:02:17.602 ********* 2021-03-09T08:51:12.566 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T08:51:12.568 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.594 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.034) 0:02:17.636 ********* 2021-03-09T08:51:12.602 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.631 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.036) 0:02:17.673 ********* 2021-03-09T08:51:12.638 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:12.640 INFO:teuthology.task.ansible.out:TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.663 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.032) 0:02:17.705 ********* 2021-03-09T08:51:12.670 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.698 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.035) 0:02:17.740 ********* 2021-03-09T08:51:12.705 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:51:12.706 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.732 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.033) 0:02:17.774 ********* 2021-03-09T08:51:12.739 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.771 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.039) 0:02:17.813 ********* 2021-03-09T08:51:12.778 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T08:51:12.780 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.037) 0:02:17.851 ********* 2021-03-09T08:51:12.816 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:12.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.032) 0:02:17.884 ********* 2021-03-09T08:51:12.944 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T08:51:12.945 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:51:12.969 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:12 +0000 (0:00:00.126) 0:02:18.011 ********* 2021-03-09T08:51:12.988 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:51:13.013 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.043) 0:02:18.055 ********* 2021-03-09T08:51:13.019 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:51:13.021 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'block_used': 13643752, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451356, 'size_available': 927544754176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113942, 'block_size': 4096, 'inode_available': 60941034}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451356, "block_size": 4096, "block_total": 240095108, "block_used": 13643752, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941034, "inode_total": 61054976, "inode_used": 113942, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544754176, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.046 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.033) 0:02:18.088 ********* 2021-03-09T08:51:13.064 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'block_used': 13643752, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451356, 'size_available': 927544754176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113942, 'block_size': 4096, 'inode_available': 60941034}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451356, "block_size": 4096, "block_total": 240095108, "block_used": 13643752, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941034, "inode_total": 61054976, "inode_used": 113942, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544754176, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.092 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.045) 0:02:18.133 ********* 2021-03-09T08:51:13.099 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T08:51:13.100 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.227 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.135) 0:02:18.269 ********* 2021-03-09T08:51:13.237 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.264 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.036) 0:02:18.306 ********* 2021-03-09T08:51:13.270 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:13.272 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.298 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.034) 0:02:18.340 ********* 2021-03-09T08:51:13.306 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.333 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.034) 0:02:18.375 ********* 2021-03-09T08:51:13.340 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:13.342 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.034) 0:02:18.409 ********* 2021-03-09T08:51:13.376 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.401 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.033) 0:02:18.442 ********* 2021-03-09T08:51:13.407 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T08:51:13.409 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.433 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.032) 0:02:18.475 ********* 2021-03-09T08:51:13.440 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.466 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.032) 0:02:18.507 ********* 2021-03-09T08:51:13.475 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T08:51:13.477 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.502 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.036) 0:02:18.544 ********* 2021-03-09T08:51:13.510 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.537 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.035) 0:02:18.579 ********* 2021-03-09T08:51:13.544 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T08:51:13.546 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.574 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.036) 0:02:18.615 ********* 2021-03-09T08:51:13.582 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:13.616 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:00.041) 0:02:18.657 ********* 2021-03-09T08:51:14.074 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T08:51:14.076 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T08:51:14.106 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:14 +0000 (0:00:00.490) 0:02:19.148 ********* 2021-03-09T08:51:14.115 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T08:51:14.145 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:14 +0000 (0:00:00.039) 0:02:19.187 ********* 2021-03-09T08:51:14.564 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:51:14.565 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:14.596 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:14.797 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:14.852 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.041 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.085 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.296 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.349 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.539 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.570 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:01.425) 0:02:20.612 ********* 2021-03-09T08:51:15.602 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:51:15.631 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.060) 0:02:20.672 ********* 2021-03-09T08:51:15.724 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:51:15.725 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T08:51:15.753 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.122) 0:02:20.795 ********* 2021-03-09T08:51:15.782 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T08:51:15.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.055) 0:02:20.851 ********* 2021-03-09T08:51:15.932 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:51:15.934 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:15.986 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.061 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.104 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.178 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.253 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.317 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.332 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.557) 0:02:21.409 ********* 2021-03-09T08:51:16.387 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:51:16.411 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.412 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:51:16.439 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.071) 0:02:21.480 ********* 2021-03-09T08:51:16.460 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.021) 0:02:21.502 ********* 2021-03-09T08:51:16.477 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T08:51:16.479 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.501 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.040) 0:02:21.542 ********* 2021-03-09T08:51:16.502 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:16.509 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.535 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.034) 0:02:21.577 ********* 2021-03-09T08:51:16.562 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T08:51:16.564 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.586 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.051) 0:02:21.628 ********* 2021-03-09T08:51:16.634 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T08:51:16.636 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:16.663 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.076) 0:02:21.704 ********* 2021-03-09T08:51:16.702 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T08:51:16.703 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T08:51:16.728 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.065) 0:02:21.770 ********* 2021-03-09T08:51:16.736 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T08:51:16.738 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:16.763 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.034) 0:02:21.805 ********* 2021-03-09T08:51:16.803 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T08:51:16.805 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T08:51:16.834 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.070) 0:02:21.875 ********* 2021-03-09T08:51:17.124 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-09T08:51:17.125 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true} 2021-03-09T08:51:17.155 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.320) 0:02:22.196 ********* 2021-03-09T08:51:17.298 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true} 2021-03-09T08:51:17.323 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.168) 0:02:22.365 ********* 2021-03-09T08:51:17.490 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T08:51:17.492 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T08:51:17.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.193) 0:02:22.559 ********* 2021-03-09T08:51:17.526 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T08:51:17.527 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:17.557 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.040) 0:02:22.599 ********* 2021-03-09T08:51:17.559 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:17.567 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:17.569 INFO:teuthology.task.ansible.out:TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.596 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.038) 0:02:22.638 ********* 2021-03-09T08:51:17.604 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.633 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.037) 0:02:22.675 ********* 2021-03-09T08:51:17.652 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-09T08:51:17.654 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.678 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.045) 0:02:22.720 ********* 2021-03-09T08:51:17.685 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.712 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.033) 0:02:22.754 ********* 2021-03-09T08:51:17.719 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:51:17.720 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.755 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.043) 0:02:22.797 ********* 2021-03-09T08:51:17.763 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.788 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.032) 0:02:22.830 ********* 2021-03-09T08:51:17.795 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:51:17.797 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.032) 0:02:22.862 ********* 2021-03-09T08:51:17.828 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.865 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.045) 0:02:22.907 ********* 2021-03-09T08:51:18.000 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-09T08:51:18.002 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:51:18.033 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.167) 0:02:23.075 ********* 2021-03-09T08:51:18.041 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:51:18.068 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.035) 0:02:23.110 ********* 2021-03-09T08:51:18.070 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:18.195 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-09T08:51:18.196 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:18.223 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.151) 0:02:23.262 ********* 2021-03-09T08:51:18.228 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:18.255 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.034) 0:02:23.297 ********* 2021-03-09T08:51:18.589 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T08:51:18.591 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T08:51:18.614 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.359) 0:02:23.656 ********* 2021-03-09T08:51:18.622 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T08:51:18.650 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.035) 0:02:23.692 ********* 2021-03-09T08:51:18.976 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T08:51:18.978 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T08:51:19.001 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.351) 0:02:24.043 ********* 2021-03-09T08:51:19.019 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T08:51:19.045 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:19 +0000 (0:00:00.043) 0:02:24.086 ********* 2021-03-09T08:51:19.987 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:51:19.995 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:51:20.021 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.976) 0:02:25.063 ********* 2021-03-09T08:51:20.048 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.026) 0:02:25.090 ********* 2021-03-09T08:51:20.249 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T08:51:20.250 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046793", "end": "2021-03-09 08:51:20.205207", "rc": 0, "start": "2021-03-09 08:51:20.158414", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-09T08:51:20.273 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.224) 0:02:25.315 ********* 2021-03-09T08:51:20.281 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048884", "end": "2021-03-09 08:51:20.042333", "rc": 0, "start": "2021-03-09 08:51:19.993449", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-09T08:51:20.308 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.034) 0:02:25.349 ********* 2021-03-09T08:51:20.315 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-09T08:51:20.317 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.038) 0:02:25.388 ********* 2021-03-09T08:51:20.357 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.389 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.043) 0:02:25.431 ********* 2021-03-09T08:51:20.399 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T08:51:20.401 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.435 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.045) 0:02:25.477 ********* 2021-03-09T08:51:20.446 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.475 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.039) 0:02:25.517 ********* 2021-03-09T08:51:21.360 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-09T08:51:21.362 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:51:21.386 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.910) 0:02:26.427 ********* 2021-03-09T08:51:21.399 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:51:21.424 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.038) 0:02:26.466 ********* 2021-03-09T08:51:21.431 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T08:51:21.433 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.461 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.036) 0:02:26.503 ********* 2021-03-09T08:51:21.468 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.495 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.033) 0:02:26.536 ********* 2021-03-09T08:51:21.827 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T08:51:21.829 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T08:51:21.853 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.357) 0:02:26.894 ********* 2021-03-09T08:51:21.862 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T08:51:21.889 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.036) 0:02:26.930 ********* 2021-03-09T08:51:21.897 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T08:51:21.899 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.924 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.035) 0:02:26.966 ********* 2021-03-09T08:51:21.932 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.961 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.036) 0:02:27.002 ********* 2021-03-09T08:51:22.127 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T08:51:22.128 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ActiveEnterTimestampMonotonic": "23803918", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target -.mount systemd-tmpfiles-setup.service system.slice network.target systemd-journald.socket basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:04 UTC", "AssertTimestampMonotonic": "17256203", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:04 UTC", "ConditionTimestampMonotonic": "17256203", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-09T08:51:22.130 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1993", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ExecMainStartTimestampMonotonic": "23803866", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:04 UTC", "InactiveExitTimestampMonotonic": "17257030", "InvocationID": "277583e46dbf46d1b1952bd60e5bde10", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-09T08:51:22.132 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1993", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:11 UTC", "S 2021-03-09T08:51:22.133 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23803918", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:48:11 UTC", "WatchdogTimestampMonotonic": "23803915", "WatchdogUSec": "0"}} 2021-03-09T08:51:22.155 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.194) 0:02:27.197 ********* 2021-03-09T08:51:22.157 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:22.178 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:17 UTC", "ActiveEnterTimestampMonotonic": "23142780", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket system.slice systemd-tmpfiles-setup.service sysinit.target basic.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:11 UTC", "AssertTimestampMonotonic": "16592153", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:11 UTC", "ConditionTimestampMonotonic": "16592152", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-09T08:51:22.180 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2147", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:17 UTC", "ExecMainStartTimestampMonotonic": "23142768", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:11 UTC", "InactiveExitTimestampMonotonic": "16592908", "InvocationID": "86815df76cfa43208e33e715336582fe", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-09T08:51:22.182 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2147", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:17 UTC", "S 2021-03-09T08:51:22.183 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23142780", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:48:17 UTC", "WatchdogTimestampMonotonic": "23142779", "WatchdogUSec": "0"}} TASK [testnode : set_fact] ***************************************************** ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T08:51:22.208 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.052) 0:02:27.250 ********* 2021-03-09T08:51:22.231 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.022) 0:02:27.273 ********* 2021-03-09T08:51:22.238 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:22.240 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.265 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.033) 0:02:27.306 ********* 2021-03-09T08:51:22.271 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:22.273 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T08:51:22.300 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.035) 0:02:27.342 ********* 2021-03-09T08:51:22.307 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-09T08:51:22.309 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.333 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.032) 0:02:27.374 ********* 2021-03-09T08:51:22.340 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:22.341 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.369 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.036) 0:02:27.411 ********* 2021-03-09T08:51:22.407 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-09T08:51:22.409 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.433 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.064) 0:02:27.475 ********* 2021-03-09T08:51:23.229 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T08:51:23.231 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:51:23.254 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.820) 0:02:28.296 ********* 2021-03-09T08:51:23.281 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T08:51:23.283 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.305 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.051) 0:02:28.347 ********* 2021-03-09T08:51:23.335 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T08:51:23.336 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:51:23.360 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.054) 0:02:28.402 ********* 2021-03-09T08:51:23.388 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T08:51:23.389 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.413 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.053) 0:02:28.455 ********* 2021-03-09T08:51:23.442 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T08:51:23.444 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:51:23.468 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.054) 0:02:28.509 ********* 2021-03-09T08:51:23.559 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:51:23.585 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.117) 0:02:28.627 ********* 2021-03-09T08:51:23.848 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-09T08:51:23.850 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T08:51:23.877 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.291) 0:02:28.919 ********* 2021-03-09T08:51:23.879 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:23.957 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T08:51:23.985 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.107) 0:02:29.026 ********* 2021-03-09T08:51:24.980 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T08:51:25.005 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:01.020) 0:02:30.047 ********* 2021-03-09T08:51:25.045 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T08:51:25.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.066) 0:02:30.113 ********* 2021-03-09T08:51:25.611 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T08:51:25.613 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false} 2021-03-09T08:51:25.635 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.563) 0:02:30.676 ********* 2021-03-09T08:51:25.646 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false} 2021-03-09T08:51:25.674 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.039) 0:02:30.716 ********* 2021-03-09T08:51:25.682 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T08:51:25.685 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-09T08:51:25.717 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.041) 0:02:30.757 ********* 2021-03-09T08:51:25.723 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-09T08:51:25.758 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.043) 0:02:30.800 ********* 2021-03-09T08:51:25.863 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T08:51:25.865 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615279700.4209394, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615279700.4169414, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615279700.4169414, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T08:51:25.891 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.132) 0:02:30.932 ********* 2021-03-09T08:51:25.920 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615279708.4119506, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615279706.9071987, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615279706.9071987, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T08:51:25.946 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.055) 0:02:30.988 ********* 2021-03-09T08:51:26.231 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T08:51:26.233 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T08:51:26.257 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.310) 0:02:31.298 ********* 2021-03-09T08:51:26.279 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.022) 0:02:31.321 ********* 2021-03-09T08:51:26.297 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T08:51:26.299 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T08:51:26.333 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.051) 0:02:31.373 ********* 2021-03-09T08:51:26.339 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T08:51:26.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.033) 0:02:31.406 ********* 2021-03-09T08:51:26.371 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-09T08:51:26.373 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.034) 0:02:31.441 ********* 2021-03-09T08:51:26.418 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.445 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.043) 0:02:31.484 ********* 2021-03-09T08:51:26.555 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T08:51:26.593 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.148) 0:02:31.633 ********* 2021-03-09T08:51:26.603 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T08:51:26.639 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.047) 0:02:31.681 ********* 2021-03-09T08:51:26.748 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T08:51:26.786 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.147) 0:02:31.828 ********* 2021-03-09T08:51:26.795 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T08:51:26.822 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.036) 0:02:31.864 ********* 2021-03-09T08:51:26.832 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T08:51:26.833 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.861 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.038) 0:02:31.903 ********* 2021-03-09T08:51:26.870 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.898 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.035) 0:02:31.939 ********* 2021-03-09T08:51:26.920 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi038.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.922 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.931 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.966 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.068) 0:02:32.007 ********* 2021-03-09T08:51:26.974 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.976 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.987 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:26.989 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.017 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.051) 0:02:32.058 ********* 2021-03-09T08:51:27.033 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.035 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.037 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.038 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.054) 0:02:32.113 ********* 2021-03-09T08:51:27.090 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.092 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.103 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.105 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:27.107 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.132 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.059) 0:02:32.173 ********* 2021-03-09T08:51:27.160 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.027) 0:02:32.201 ********* 2021-03-09T08:51:27.161 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:27.168 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T08:51:27.169 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.194 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.035) 0:02:32.236 ********* 2021-03-09T08:51:27.202 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.232 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.037) 0:02:32.274 ********* 2021-03-09T08:51:27.234 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:27.242 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T08:51:27.244 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.270 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.038) 0:02:32.312 ********* 2021-03-09T08:51:27.278 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T08:51:27.280 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.308 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.037) 0:02:32.350 ********* 2021-03-09T08:51:27.316 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T08:51:27.318 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.342 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.033) 0:02:32.384 ********* 2021-03-09T08:51:27.350 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T08:51:27.352 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.039) 0:02:32.423 ********* 2021-03-09T08:51:27.389 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T08:51:27.390 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.417 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.035) 0:02:32.459 ********* 2021-03-09T08:51:27.425 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T08:51:27.426 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.452 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.034) 0:02:32.494 ********* 2021-03-09T08:51:27.463 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:51:27.467 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.494 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.042) 0:02:32.536 ********* 2021-03-09T08:51:27.505 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T08:51:27.507 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.535 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.040) 0:02:32.576 ********* 2021-03-09T08:51:27.545 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T08:51:27.546 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.589 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.054) 0:02:32.631 ********* 2021-03-09T08:51:27.596 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:51:27.598 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.623 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.034) 0:02:32.665 ********* 2021-03-09T08:51:27.630 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:51:27.632 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.656 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.033) 0:02:32.698 ********* 2021-03-09T08:51:27.663 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T08:51:27.665 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.693 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.036) 0:02:32.735 ********* 2021-03-09T08:51:27.722 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:51:27.723 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:27.748 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.055) 0:02:32.790 ********* 2021-03-09T08:51:27.828 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T08:51:27.830 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009678", "end": "2021-03-09 08:51:27.766037", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 08:51:27.756359", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:27.935 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011096", "end": "2021-03-09 08:51:27.697743", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 08:51:27.686647", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:27.946 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006092", "end": "2021-03-09 08:51:27.890815", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 08:51:27.884723", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:28.061 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006012", "end": "2021-03-09 08:51:27.843559", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 08:51:27.837547", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:28.072 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005479", "end": "2021-03-09 08:51:28.020353", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:51:28.014874", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:28.177 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005497", "end": "2021-03-09 08:51:27.964322", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:51:27.958825", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:28.220 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025374", "end": "2021-03-09 08:51:28.163063", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 08:51:28.137689", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:28.247 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.498) 0:02:33.288 ********* 2021-03-09T08:51:28.276 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T08:51:28.301 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.054) 0:02:33.343 ********* 2021-03-09T08:51:28.320 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T08:51:28.322 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015987", "end": "2021-03-09 08:51:28.092090", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 08:51:28.076103", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:28.351 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.049) 0:02:33.393 ********* 2021-03-09T08:51:28.378 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T08:51:28.403 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.051) 0:02:33.445 ********* 2021-03-09T08:51:28.441 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T08:51:28.443 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:51:28.544 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:51:28.614 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-09T08:51:28.635 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.232) 0:02:33.677 ********* 2021-03-09T08:51:28.660 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.024) 0:02:33.702 ********* 2021-03-09T08:51:29.434 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi041.front.sepia.ceph.com] 2021-03-09T08:51:29.487 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.827) 0:02:34.529 ********* 2021-03-09T08:51:29.538 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T08:51:29.566 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.075) 0:02:34.605 ********* 2021-03-09T08:51:29.601 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:29.603 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.626 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.063) 0:02:34.668 ********* 2021-03-09T08:51:29.686 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-09T08:51:29.688 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T08:51:29.712 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.086) 0:02:34.754 ********* 2021-03-09T08:51:30.401 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi038.front.sepia.ceph.com] 2021-03-09T08:51:30.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.742) 0:02:35.497 ********* 2021-03-09T08:51:30.494 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:30.496 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T08:51:30.518 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.062) 0:02:35.560 ********* 2021-03-09T08:51:30.545 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:30.547 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.569 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.050) 0:02:35.611 ********* 2021-03-09T08:51:30.619 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T08:51:30.641 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.072) 0:02:35.683 ********* 2021-03-09T08:51:30.661 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T08:51:30.662 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"cache_update_time": 1615279725, "cache_updated": false, "changed": false} 2021-03-09T08:51:30.689 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.048) 0:02:35.731 ********* 2021-03-09T08:51:30.727 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:30.729 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.753 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.064) 0:02:35.795 ********* 2021-03-09T08:51:30.792 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:30.794 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T08:51:30.817 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.063) 0:02:35.859 ********* 2021-03-09T08:51:31.006 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T08:51:31.008 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:31.134 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:31.180 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.363) 0:02:36.222 ********* 2021-03-09T08:51:31.182 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:31.583 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T08:51:31.585 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279891.2577248-15546-41869293061123/source", "state": "file", "uid": 0} 2021-03-09T08:51:31.605 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T08:51:31.607 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1615279721, "cache_updated": false, "changed": false} 2021-03-09T08:51:31.632 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.451) 0:02:36.674 ********* 2021-03-09T08:51:31.674 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:31.676 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.698 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.066) 0:02:36.740 ********* 2021-03-09T08:51:31.737 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:51:31.738 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T08:51:31.762 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.063) 0:02:36.804 ********* 2021-03-09T08:51:31.927 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T08:51:31.929 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:31.949 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T08:51:31.951 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279891.6304283-15546-182595277479316/source", "state": "file", "uid": 0} 2021-03-09T08:51:31.978 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.216) 0:02:37.020 ********* 2021-03-09T08:51:32.050 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T08:51:32.051 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:32.078 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.099) 0:02:37.120 ********* 2021-03-09T08:51:32.469 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T08:51:32.471 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279892.150487-15687-93266415837031/source", "state": "file", "uid": 0} 2021-03-09T08:51:32.809 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279892.4990807-15687-104975068029517/source", "state": "file", "uid": 0} 2021-03-09T08:51:32.835 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.756) 0:02:37.877 ********* 2021-03-09T08:51:35.706 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T08:51:35.708 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qNJ7uj-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qNJ7uj-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-09T08:51:35.710 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-09T08:51:35.732 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:35 +0000 (0:00:02.897) 0:02:40.774 ********* 2021-03-09T08:51:35.891 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T08:51:35.893 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T08:51:35.930 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:35 +0000 (0:00:00.197) 0:02:40.972 ********* 2021-03-09T08:51:36.302 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-09T08:51:36.304 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279895.991896-16167-28872924646966/source", "state": "file", "uid": 0} 2021-03-09T08:51:36.331 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.400) 0:02:41.372 ********* 2021-03-09T08:51:36.588 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T08:51:36.590 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-vTRj98-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-vTRj98-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-09T08:51:36.591 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-09T08:51:36.616 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.285) 0:02:41.658 ********* 2021-03-09T08:51:36.622 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T08:51:36.624 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.122869", "end": "2021-03-09 08:51:36.404931", "rc": 0, "start": "2021-03-09 08:51:36.282062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:36.652 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.036) 0:02:41.694 ********* 2021-03-09T08:51:36.755 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T08:51:36.757 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T08:51:36.782 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.129) 0:02:41.823 ********* 2021-03-09T08:51:37.163 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-09T08:51:37.165 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279896.8421986-16272-194641159524073/source", "state": "file", "uid": 0} 2021-03-09T08:51:37.187 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.405) 0:02:42.229 ********* 2021-03-09T08:51:37.457 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:37.458 INFO:teuthology.task.ansible.out:TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T08:51:37.459 INFO:teuthology.task.ansible.out: changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.119752", "end": "2021-03-09 08:51:37.403086", "rc": 0, "start": "2021-03-09 08:51:37.283334", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:37.485 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.298) 0:02:42.527 ********* 2021-03-09T08:51:39.854 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-09T08:51:39.855 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network-online.target basic.target firewalld.service docker.socket sysinit.target containerd.service systemd-journald.socket system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-09T08:51:39.857 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-03-09T08:51:39.859 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice docker.socket sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:33 UTC", "StateChangeTimestampMonotonic": "39134919", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-09T08:51:39.861 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:51:40.917 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket docker.socket firewalld.service containerd.service sysinit.target basic.target network-online.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-09T08:51:40.919 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-03-09T08:51:40.920 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:27 UTC", "StateChangeTimestampMonotonic": "40280689", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-09T08:51:40.922 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:51:40.990 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-09T08:51:40.991 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-09T08:51:40.993 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi038.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi041.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Tuesday 09 March 2021 08:51:40 +0000 (0:00:03.507) 0:02:46.034 ********* =============================================================================== 2021-03-09T08:51:40.993 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 21.37s users : Update authorized_keys using the keys repo --------------------- 18.76s testnode : Ensure packages are not present. ----------------------------- 9.61s testnode : Install packages --------------------------------------------- 8.74s testnode : Zap all non-root disks --------------------------------------- 8.27s Remove revoked users ---------------------------------------------------- 6.41s container-host : Restart docker service --------------------------------- 3.51s 2021-03-09T08:51:40.994 INFO:teuthology.task.ansible.out:container-host : Install registries-conf-ctl ---------------------------- 2.90s testnode : Update apt cache. -------------------------------------------- 2.68s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.95s testnode : Install apt keys --------------------------------------------- 1.78s Gathering Facts --------------------------------------------------------- 1.69s common : Update apt cache ----------------------------------------------- 1.68s testnode : Create logical volume(s) ------------------------------------- 1.43s testnode : Install packages via pip ------------------------------------- 1.02s testnode : Make sure apt dependencies are installed --------------------- 1.01s common : Set the hardware clock ----------------------------------------- 0.99s testnode : Ensu 2021-03-09T08:51:40.995 INFO:teuthology.task.ansible.out:re perl-doc is installed on apt systems. ----------------- 0.98s testnode : Install non aarch64 packages --------------------------------- 0.92s testnode : Install ntp package on apt based systems. -------------------- 0.91s 2021-03-09T08:51:41.380 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi041.front.sepia.ceph.com')] 2021-03-09T08:51:41.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:51:41.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:51:41.459 DEBUG:teuthology.orchestra.run.smithi038:> true 2021-03-09T08:51:41.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:51:41.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:51:41.734 DEBUG:teuthology.orchestra.run.smithi041:> true 2021-03-09T08:51:41.939 INFO:teuthology.run_tasks:Running task clock... 2021-03-09T08:51:41.955 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-09T08:51:41.956 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T08:51:41.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T08:51:42.019 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-09T08:51:42.020 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Command line: ntpd -gq 2021-03-09T08:51:42.020 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: proto: precision = 0.123 usec (-23) 2021-03-09T08:51:42.021 INFO:teuthology.orchestra.run.smithi038.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:51:42.022 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:51:42.022 INFO:teuthology.orchestra.run.smithi038.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:51:42.023 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:51:42.024 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T08:51:42.024 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T08:51:42.024 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T08:51:42.024 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listen normally on 3 ens1f0 172.21.15.38:123 2021-03-09T08:51:42.024 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listen normally on 4 lo [::1]:123 2021-03-09T08:51:42.025 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15fc%4]:123 2021-03-09T08:51:42.025 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:41 ntpd[9509]: Listening on routing socket on fd #22 for interface updates 2021-03-09T08:51:43.020 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:42 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:43.020 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:42 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:43.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:42 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:43.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:42 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:45.020 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:44 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:45.020 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:44 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:45.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:44 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:45.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:44 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:46.020 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:45 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:46.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:45 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:46.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:45 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:46.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:45 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:47.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:47.022 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:47.022 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:47.022 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:47.023 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:47.023 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:47.023 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:47.023 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:47.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:47.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:47.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:47.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:46 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:48.020 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:47 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:48.021 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:47 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:48.022 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:47 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:48.022 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:47 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:49.022 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:49.023 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:49.023 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:49.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:49.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:49.024 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:49.025 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:49.025 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:48 ntpd[9509]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: 9 Mar 08:51:49 ntpd[9509]: ntpd: time set +0.026744 s 2021-03-09T08:51:49.028 INFO:teuthology.orchestra.run.smithi038.stdout:ntpd: time set +0.026744s 2021-03-09T08:51:49.029 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:49 ntpd[9509]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-09T08:51:49.029 INFO:teuthology.orchestra.run.smithi038.stderr: 9 Mar 08:51:49 ntpd[9509]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T08:51:49.069 INFO:teuthology.orchestra.run.smithi038.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T08:51:49.069 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================== 2021-03-09T08:51:49.070 INFO:teuthology.orchestra.run.smithi038.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:49.070 INFO:teuthology.orchestra.run.smithi038.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:49.070 INFO:teuthology.orchestra.run.smithi038.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:49.071 INFO:teuthology.orchestra.run.smithi038.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:49.071 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T08:51:49.075 DEBUG:teuthology.orchestra.run.smithi041:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T08:51:49.141 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-09T08:51:49.142 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Command line: ntpd -gq 2021-03-09T08:51:49.142 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: proto: precision = 0.127 usec (-23) 2021-03-09T08:51:49.143 INFO:teuthology.orchestra.run.smithi041.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:51:49.144 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:51:49.144 INFO:teuthology.orchestra.run.smithi041.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:51:49.145 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:51:49.146 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T08:51:49.146 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T08:51:49.146 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T08:51:49.146 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listen normally on 3 ens1f0 172.21.15.41:123 2021-03-09T08:51:49.147 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listen normally on 4 lo [::1]:123 2021-03-09T08:51:49.147 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1252%3]:123 2021-03-09T08:51:49.147 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:48 ntpd[9644]: Listening on routing socket on fd #22 for interface updates 2021-03-09T08:51:50.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:50.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:50.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:49 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:52.141 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:52.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:52.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:52.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:52.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:52.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:52.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:52.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:52.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:52.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:52.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:52.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:51 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:54.141 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:54.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:54.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:54.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:54.142 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:54.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:54.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:54.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:54.143 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:54.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:54.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:54.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:54.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:54.144 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:54.145 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:54.145 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:53 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:56.139 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:55 ntpd[9644]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:51:56.139 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:55 ntpd[9644]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:51:56.139 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:55 ntpd[9644]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:51:56.139 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:55 ntpd[9644]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:51:56.140 INFO:teuthology.orchestra.run.smithi041.stdout: 9 Mar 08:51:56 ntpd[9644]: ntpd: time set +0.175353 s 2021-03-09T08:51:56.140 INFO:teuthology.orchestra.run.smithi041.stdout:ntpd: time set +0.175353s 2021-03-09T08:51:56.141 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:56 ntpd[9644]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-09T08:51:56.141 INFO:teuthology.orchestra.run.smithi041.stderr: 9 Mar 08:51:56 ntpd[9644]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T08:51:56.191 INFO:teuthology.orchestra.run.smithi041.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T08:51:56.191 INFO:teuthology.orchestra.run.smithi041.stdout:============================================================================== 2021-03-09T08:51:56.191 INFO:teuthology.orchestra.run.smithi041.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:56.192 INFO:teuthology.orchestra.run.smithi041.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:56.192 INFO:teuthology.orchestra.run.smithi041.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:56.192 INFO:teuthology.orchestra.run.smithi041.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:51:56.193 INFO:teuthology.run_tasks:Running task install... 2021-03-09T08:51:56.204 DEBUG:teuthology.task.install:project ceph 2021-03-09T08:51:56.207 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}} 2021-03-09T08:51:56.207 DEBUG:teuthology.task.install:config {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'} 2021-03-09T08:51:56.208 INFO:teuthology.task.install:Using flavor: basic 2021-03-09T08:51:56.216 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-09T08:51:56.218 INFO:teuthology.task.install:extra packages: [] 2021-03-09T08:51:56.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo lsb_release -is 2021-03-09T08:51:56.273 INFO:teuthology.orchestra.run.smithi038.stdout:Ubuntu 2021-03-09T08:51:56.278 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:51:56.279 DEBUG:teuthology.orchestra.run.smithi041:> sudo lsb_release -is 2021-03-09T08:51:56.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-key list | grep Ceph 2021-03-09T08:51:56.335 INFO:teuthology.orchestra.run.smithi041.stdout:Ubuntu 2021-03-09T08:51:56.340 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:51:56.341 DEBUG:teuthology.orchestra.run.smithi041:> sudo apt-key list | grep Ceph 2021-03-09T08:51:56.378 INFO:teuthology.orchestra.run.smithi038.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T08:51:56.435 INFO:teuthology.orchestra.run.smithi038.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T08:51:56.436 INFO:teuthology.orchestra.run.smithi038.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T08:51:56.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo lsb_release -is 2021-03-09T08:51:56.439 INFO:teuthology.orchestra.run.smithi041.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T08:51:56.489 INFO:teuthology.orchestra.run.smithi038.stdout:Ubuntu 2021-03-09T08:51:56.494 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:51:56.495 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T08:51:56.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:51:56.505 INFO:teuthology.orchestra.run.smithi041.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T08:51:56.506 INFO:teuthology.orchestra.run.smithi041.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T08:51:56.506 DEBUG:teuthology.orchestra.run.smithi041:> sudo lsb_release -is 2021-03-09T08:51:56.556 INFO:teuthology.orchestra.run.smithi041.stdout:Ubuntu 2021-03-09T08:51:56.561 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:51:56.562 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T08:51:56.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:51:56.666 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default/ 2021-03-09T08:51:56.666 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:51:56.711 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default/ 2021-03-09T08:51:56.711 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:51:56.789 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:51:56.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T08:51:56.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2021-03-09T08:51:56.830 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:51:56.830 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T08:51:56.847 DEBUG:teuthology.orchestra.run.smithi041:> sudo apt-get update 2021-03-09T08:51:56.991 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-09T08:51:57.022 INFO:teuthology.orchestra.run.smithi041.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-09T08:51:57.053 INFO:teuthology.orchestra.run.smithi038.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic InRelease 2021-03-09T08:51:57.077 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-09T08:51:57.090 INFO:teuthology.orchestra.run.smithi041.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic InRelease 2021-03-09T08:51:57.101 INFO:teuthology.orchestra.run.smithi038.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-09T08:51:57.102 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T08:51:57.113 INFO:teuthology.orchestra.run.smithi041.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-09T08:51:57.124 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-03-09T08:51:57.138 INFO:teuthology.orchestra.run.smithi041.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-09T08:51:57.139 INFO:teuthology.orchestra.run.smithi041.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T08:51:57.161 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,099 B] 2021-03-09T08:51:57.162 INFO:teuthology.orchestra.run.smithi041.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,099 B] 2021-03-09T08:51:57.180 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-09T08:51:57.195 INFO:teuthology.orchestra.run.smithi041.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-03-09T08:51:57.218 INFO:teuthology.orchestra.run.smithi041.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-09T08:51:57.260 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-09T08:51:57.298 INFO:teuthology.orchestra.run.smithi041.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-09T08:51:57.601 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 27.4 kB in 1s (46.2 kB/s) 2021-03-09T08:51:57.639 INFO:teuthology.orchestra.run.smithi041.stdout:Fetched 27.4 kB in 1s (46.1 kB/s) 2021-03-09T08:51:58.896 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2021-03-09T08:51:58.934 INFO:teuthology.orchestra.run.smithi041.stdout:Reading package lists... 2021-03-09T08:51:58.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1bionic cephadm=17.0.0-1742-g89fb622b-1bionic ceph-mds=17.0.0-1742-g89fb622b-1bionic ceph-mgr=17.0.0-1742-g89fb622b-1bionic ceph-common=17.0.0-1742-g89fb622b-1bionic ceph-fuse=17.0.0-1742-g89fb622b-1bionic ceph-test=17.0.0-1742-g89fb622b-1bionic radosgw=17.0.0-1742-g89fb622b-1bionic python3-rados=17.0.0-1742-g89fb622b-1bionic python3-rgw=17.0.0-1742-g89fb622b-1bionic python3-cephfs=17.0.0-1742-g89fb622b-1bionic python3-rbd=17.0.0-1742-g89fb622b-1bionic libcephfs2=17.0.0-1742-g89fb622b-1bionic libcephfs-dev=17.0.0-1742-g89fb622b-1bionic librados2=17.0.0-1742-g89fb622b-1bionic librbd1=17.0.0-1742-g89fb622b-1bionic rbd-fuse=17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:51:58.984 DEBUG:teuthology.orchestra.run.smithi041:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1bionic cephadm=17.0.0-1742-g89fb622b-1bionic ceph-mds=17.0.0-1742-g89fb622b-1bionic ceph-mgr=17.0.0-1742-g89fb622b-1bionic ceph-common=17.0.0-1742-g89fb622b-1bionic ceph-fuse=17.0.0-1742-g89fb622b-1bionic ceph-test=17.0.0-1742-g89fb622b-1bionic radosgw=17.0.0-1742-g89fb622b-1bionic python3-rados=17.0.0-1742-g89fb622b-1bionic python3-rgw=17.0.0-1742-g89fb622b-1bionic python3-cephfs=17.0.0-1742-g89fb622b-1bionic python3-rbd=17.0.0-1742-g89fb622b-1bionic libcephfs2=17.0.0-1742-g89fb622b-1bionic libcephfs-dev=17.0.0-1742-g89fb622b-1bionic librados2=17.0.0-1742-g89fb622b-1bionic librbd1=17.0.0-1742-g89fb622b-1bionic rbd-fuse=17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:51:59.033 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2021-03-09T08:51:59.071 INFO:teuthology.orchestra.run.smithi041.stdout:Reading package lists... 2021-03-09T08:51:59.299 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2021-03-09T08:51:59.300 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2021-03-09T08:51:59.339 INFO:teuthology.orchestra.run.smithi041.stdout:Building dependency tree... 2021-03-09T08:51:59.340 INFO:teuthology.orchestra.run.smithi041.stdout:Reading state information... 2021-03-09T08:51:59.479 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2021-03-09T08:51:59.479 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T08:51:59.481 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T08:51:59.481 INFO:teuthology.orchestra.run.smithi038.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-09T08:51:59.482 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-09T08:51:59.482 INFO:teuthology.orchestra.run.smithi038.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-09T08:51:59.482 INFO:teuthology.orchestra.run.smithi038.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T08:51:59.482 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-09T08:51:59.483 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-09T08:51:59.483 INFO:teuthology.orchestra.run.smithi038.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-09T08:51:59.483 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-09T08:51:59.483 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T08:51:59.483 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-09T08:51:59.484 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-09T08:51:59.484 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-09T08:51:59.484 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-09T08:51:59.484 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-09T08:51:59.485 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-09T08:51:59.485 INFO:teuthology.orchestra.run.smithi038.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-09T08:51:59.485 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-09T08:51:59.485 INFO:teuthology.orchestra.run.smithi038.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-09T08:51:59.485 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-09T08:51:59.486 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T08:51:59.486 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2021-03-09T08:51:59.487 INFO:teuthology.orchestra.run.smithi038.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-09T08:51:59.487 INFO:teuthology.orchestra.run.smithi038.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-09T08:51:59.487 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-09T08:51:59.487 INFO:teuthology.orchestra.run.smithi038.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-09T08:51:59.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-09T08:51:59.488 INFO:teuthology.orchestra.run.smithi038.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-09T08:51:59.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-09T08:51:59.488 INFO:teuthology.orchestra.run.smithi038.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-09T08:51:59.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-09T08:51:59.489 INFO:teuthology.orchestra.run.smithi038.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-09T08:51:59.489 INFO:teuthology.orchestra.run.smithi038.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-09T08:51:59.489 INFO:teuthology.orchestra.run.smithi038.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-09T08:51:59.489 INFO:teuthology.orchestra.run.smithi038.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-09T08:51:59.490 INFO:teuthology.orchestra.run.smithi038.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-09T08:51:59.490 INFO:teuthology.orchestra.run.smithi038.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-09T08:51:59.523 INFO:teuthology.orchestra.run.smithi041.stdout:The following additional packages will be installed: 2021-03-09T08:51:59.523 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T08:51:59.525 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T08:51:59.525 INFO:teuthology.orchestra.run.smithi041.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-09T08:51:59.525 INFO:teuthology.orchestra.run.smithi041.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-09T08:51:59.526 INFO:teuthology.orchestra.run.smithi041.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-09T08:51:59.526 INFO:teuthology.orchestra.run.smithi041.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T08:51:59.526 INFO:teuthology.orchestra.run.smithi041.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-09T08:51:59.526 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-09T08:51:59.527 INFO:teuthology.orchestra.run.smithi041.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-09T08:51:59.527 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-09T08:51:59.527 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T08:51:59.527 INFO:teuthology.orchestra.run.smithi041.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-09T08:51:59.527 INFO:teuthology.orchestra.run.smithi041.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-09T08:51:59.528 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-09T08:51:59.528 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-09T08:51:59.528 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-09T08:51:59.528 INFO:teuthology.orchestra.run.smithi041.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-09T08:51:59.529 INFO:teuthology.orchestra.run.smithi041.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-09T08:51:59.529 INFO:teuthology.orchestra.run.smithi041.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-09T08:51:59.529 INFO:teuthology.orchestra.run.smithi041.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-09T08:51:59.529 INFO:teuthology.orchestra.run.smithi041.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-09T08:51:59.529 INFO:teuthology.orchestra.run.smithi041.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T08:51:59.530 INFO:teuthology.orchestra.run.smithi041.stdout:Suggested packages: 2021-03-09T08:51:59.530 INFO:teuthology.orchestra.run.smithi041.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-09T08:51:59.531 INFO:teuthology.orchestra.run.smithi041.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-09T08:51:59.531 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-09T08:51:59.531 INFO:teuthology.orchestra.run.smithi041.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-09T08:51:59.531 INFO:teuthology.orchestra.run.smithi041.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-09T08:51:59.532 INFO:teuthology.orchestra.run.smithi041.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-09T08:51:59.532 INFO:teuthology.orchestra.run.smithi041.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-09T08:51:59.532 INFO:teuthology.orchestra.run.smithi041.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-09T08:51:59.532 INFO:teuthology.orchestra.run.smithi041.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-09T08:51:59.533 INFO:teuthology.orchestra.run.smithi041.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-09T08:51:59.533 INFO:teuthology.orchestra.run.smithi041.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-09T08:51:59.533 INFO:teuthology.orchestra.run.smithi041.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-09T08:51:59.533 INFO:teuthology.orchestra.run.smithi041.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-09T08:51:59.533 INFO:teuthology.orchestra.run.smithi041.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-09T08:51:59.534 INFO:teuthology.orchestra.run.smithi041.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-09T08:51:59.590 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2021-03-09T08:51:59.591 INFO:teuthology.orchestra.run.smithi038.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T08:51:59.591 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T08:51:59.591 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T08:51:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-03-09T08:51:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-09T08:51:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-09T08:51:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-09T08:51:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T08:51:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-09T08:51:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-09T08:51:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-09T08:51:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T08:51:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T08:51:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-09T08:51:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-09T08:51:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-09T08:51:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-09T08:51:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-09T08:51:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-09T08:51:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-09T08:51:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T08:51:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-09T08:51:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-09T08:51:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-09T08:51:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T08:51:59.599 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages will be upgraded: 2021-03-09T08:51:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 librbd1 2021-03-09T08:51:59.632 INFO:teuthology.orchestra.run.smithi041.stdout:The following NEW packages will be installed: 2021-03-09T08:51:59.633 INFO:teuthology.orchestra.run.smithi041.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T08:51:59.633 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T08:51:59.635 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T08:51:59.635 INFO:teuthology.orchestra.run.smithi041.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-03-09T08:51:59.636 INFO:teuthology.orchestra.run.smithi041.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-09T08:51:59.636 INFO:teuthology.orchestra.run.smithi041.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-09T08:51:59.636 INFO:teuthology.orchestra.run.smithi041.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-09T08:51:59.636 INFO:teuthology.orchestra.run.smithi041.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T08:51:59.637 INFO:teuthology.orchestra.run.smithi041.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-09T08:51:59.637 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-09T08:51:59.637 INFO:teuthology.orchestra.run.smithi041.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-09T08:51:59.637 INFO:teuthology.orchestra.run.smithi041.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T08:51:59.637 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T08:51:59.638 INFO:teuthology.orchestra.run.smithi041.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-09T08:51:59.638 INFO:teuthology.orchestra.run.smithi041.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-09T08:51:59.638 INFO:teuthology.orchestra.run.smithi041.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-09T08:51:59.638 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-09T08:51:59.639 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-09T08:51:59.639 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-09T08:51:59.639 INFO:teuthology.orchestra.run.smithi041.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-09T08:51:59.639 INFO:teuthology.orchestra.run.smithi041.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T08:51:59.639 INFO:teuthology.orchestra.run.smithi041.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-09T08:51:59.640 INFO:teuthology.orchestra.run.smithi041.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-09T08:51:59.641 INFO:teuthology.orchestra.run.smithi041.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-09T08:51:59.641 INFO:teuthology.orchestra.run.smithi041.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T08:51:59.641 INFO:teuthology.orchestra.run.smithi041.stdout:The following packages will be upgraded: 2021-03-09T08:51:59.642 INFO:teuthology.orchestra.run.smithi041.stdout: librados2 librbd1 2021-03-09T08:51:59.734 INFO:teuthology.orchestra.run.smithi038.stdout:2 upgraded, 111 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T08:51:59.734 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 166 MB of archives. 2021-03-09T08:51:59.734 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 778 MB of additional disk space will be used. 2021-03-09T08:51:59.735 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1bionic [3,728 B] 2021-03-09T08:51:59.773 INFO:teuthology.orchestra.run.smithi041.stdout:2 upgraded, 111 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T08:51:59.773 INFO:teuthology.orchestra.run.smithi041.stdout:Need to get 166 MB of archives. 2021-03-09T08:51:59.773 INFO:teuthology.orchestra.run.smithi041.stdout:After this operation, 778 MB of additional disk space will be used. 2021-03-09T08:51:59.774 INFO:teuthology.orchestra.run.smithi041.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1bionic [3,728 B] 2021-03-09T08:51:59.779 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1bionic [3,126 kB] 2021-03-09T08:51:59.787 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-09T08:51:59.819 INFO:teuthology.orchestra.run.smithi041.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1bionic [3,126 kB] 2021-03-09T08:51:59.862 INFO:teuthology.orchestra.run.smithi041.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-09T08:51:59.869 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-09T08:51:59.942 INFO:teuthology.orchestra.run.smithi041.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-09T08:52:00.003 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1bionic [3,169 kB] 2021-03-09T08:52:00.122 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-09T08:52:00.132 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-09T08:52:00.144 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-09T08:52:00.161 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1bionic [665 kB] 2021-03-09T08:52:00.162 INFO:teuthology.orchestra.run.smithi041.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1bionic [3,169 kB] 2021-03-09T08:52:00.209 INFO:teuthology.orchestra.run.smithi041.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-09T08:52:00.213 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1bionic [340 kB] 2021-03-09T08:52:00.224 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1bionic [21.7 kB] 2021-03-09T08:52:00.225 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1bionic [163 kB] 2021-03-09T08:52:00.226 INFO:teuthology.orchestra.run.smithi041.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-09T08:52:00.230 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1bionic [29.4 kB] 2021-03-09T08:52:00.231 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1bionic [336 kB] 2021-03-09T08:52:00.248 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-09T08:52:00.251 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-09T08:52:00.270 INFO:teuthology.orchestra.run.smithi041.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-09T08:52:00.274 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1bionic [3,311 kB] 2021-03-09T08:52:00.288 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-09T08:52:00.295 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-09T08:52:00.302 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-09T08:52:00.313 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-09T08:52:00.322 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-09T08:52:00.353 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-09T08:52:00.357 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-09T08:52:00.364 INFO:teuthology.orchestra.run.smithi041.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-09T08:52:00.366 INFO:teuthology.orchestra.run.smithi041.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-09T08:52:00.368 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-09T08:52:00.381 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-09T08:52:00.388 INFO:teuthology.orchestra.run.smithi041.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-09T08:52:00.395 INFO:teuthology.orchestra.run.smithi041.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-09T08:52:00.401 INFO:teuthology.orchestra.run.smithi041.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-09T08:52:00.412 INFO:teuthology.orchestra.run.smithi041.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-09T08:52:00.432 INFO:teuthology.orchestra.run.smithi041.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-09T08:52:00.449 INFO:teuthology.orchestra.run.smithi038.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-09T08:52:00.450 INFO:teuthology.orchestra.run.smithi038.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-09T08:52:00.451 INFO:teuthology.orchestra.run.smithi038.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-09T08:52:00.453 INFO:teuthology.orchestra.run.smithi038.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-09T08:52:00.454 INFO:teuthology.orchestra.run.smithi038.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-09T08:52:00.454 INFO:teuthology.orchestra.run.smithi041.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-09T08:52:00.455 INFO:teuthology.orchestra.run.smithi038.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-09T08:52:00.457 INFO:teuthology.orchestra.run.smithi038.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-09T08:52:00.458 INFO:teuthology.orchestra.run.smithi041.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-09T08:52:00.459 INFO:teuthology.orchestra.run.smithi038.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-09T08:52:00.462 INFO:teuthology.orchestra.run.smithi041.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-09T08:52:00.476 INFO:teuthology.orchestra.run.smithi041.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-09T08:52:00.478 INFO:teuthology.orchestra.run.smithi038.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1bionic [99.3 kB] 2021-03-09T08:52:00.493 INFO:teuthology.orchestra.run.smithi041.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1bionic [665 kB] 2021-03-09T08:52:00.500 INFO:teuthology.orchestra.run.smithi041.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1bionic [340 kB] 2021-03-09T08:52:00.503 INFO:teuthology.orchestra.run.smithi041.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1bionic [21.7 kB] 2021-03-09T08:52:00.504 INFO:teuthology.orchestra.run.smithi041.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1bionic [163 kB] 2021-03-09T08:52:00.505 INFO:teuthology.orchestra.run.smithi041.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1bionic [29.4 kB] 2021-03-09T08:52:00.506 INFO:teuthology.orchestra.run.smithi041.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1bionic [336 kB] 2021-03-09T08:52:00.508 INFO:teuthology.orchestra.run.smithi038.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1bionic [414 kB] 2021-03-09T08:52:00.518 INFO:teuthology.orchestra.run.smithi038.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1bionic [21.2 MB] 2021-03-09T08:52:00.530 INFO:teuthology.orchestra.run.smithi038.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-09T08:52:00.531 INFO:teuthology.orchestra.run.smithi038.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-09T08:52:00.533 INFO:teuthology.orchestra.run.smithi041.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-09T08:52:00.533 INFO:teuthology.orchestra.run.smithi041.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-09T08:52:00.534 INFO:teuthology.orchestra.run.smithi041.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-09T08:52:00.537 INFO:teuthology.orchestra.run.smithi041.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-09T08:52:00.538 INFO:teuthology.orchestra.run.smithi041.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1bionic [3,311 kB] 2021-03-09T08:52:00.538 INFO:teuthology.orchestra.run.smithi041.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-09T08:52:00.611 INFO:teuthology.orchestra.run.smithi038.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-09T08:52:00.612 INFO:teuthology.orchestra.run.smithi038.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-09T08:52:00.612 INFO:teuthology.orchestra.run.smithi041.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-09T08:52:00.613 INFO:teuthology.orchestra.run.smithi038.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-09T08:52:00.613 INFO:teuthology.orchestra.run.smithi038.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-09T08:52:00.614 INFO:teuthology.orchestra.run.smithi041.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-09T08:52:00.617 INFO:teuthology.orchestra.run.smithi038.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-09T08:52:00.618 INFO:teuthology.orchestra.run.smithi041.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-09T08:52:00.620 INFO:teuthology.orchestra.run.smithi041.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-09T08:52:00.620 INFO:teuthology.orchestra.run.smithi041.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-09T08:52:00.625 INFO:teuthology.orchestra.run.smithi038.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-09T08:52:00.629 INFO:teuthology.orchestra.run.smithi038.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-09T08:52:00.630 INFO:teuthology.orchestra.run.smithi038.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-09T08:52:00.691 INFO:teuthology.orchestra.run.smithi041.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-09T08:52:00.693 INFO:teuthology.orchestra.run.smithi038.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-09T08:52:00.693 INFO:teuthology.orchestra.run.smithi041.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-09T08:52:00.693 INFO:teuthology.orchestra.run.smithi041.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-09T08:52:00.694 INFO:teuthology.orchestra.run.smithi041.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-09T08:52:00.761 INFO:teuthology.orchestra.run.smithi041.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-09T08:52:00.783 INFO:teuthology.orchestra.run.smithi038.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-09T08:52:00.783 INFO:teuthology.orchestra.run.smithi038.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-09T08:52:00.786 INFO:teuthology.orchestra.run.smithi041.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-09T08:52:00.793 INFO:teuthology.orchestra.run.smithi041.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-09T08:52:00.793 INFO:teuthology.orchestra.run.smithi041.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-09T08:52:00.794 INFO:teuthology.orchestra.run.smithi041.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1bionic [99.3 kB] 2021-03-09T08:52:00.795 INFO:teuthology.orchestra.run.smithi041.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-09T08:52:00.797 INFO:teuthology.orchestra.run.smithi041.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1bionic [414 kB] 2021-03-09T08:52:00.849 INFO:teuthology.orchestra.run.smithi041.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1bionic [21.2 MB] 2021-03-09T08:52:00.874 INFO:teuthology.orchestra.run.smithi041.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-09T08:52:00.874 INFO:teuthology.orchestra.run.smithi041.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-09T08:52:01.071 INFO:teuthology.orchestra.run.smithi038.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-09T08:52:01.106 INFO:teuthology.orchestra.run.smithi038.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-09T08:52:01.107 INFO:teuthology.orchestra.run.smithi038.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-09T08:52:01.144 INFO:teuthology.orchestra.run.smithi041.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-09T08:52:01.154 INFO:teuthology.orchestra.run.smithi038.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-09T08:52:01.155 INFO:teuthology.orchestra.run.smithi038.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-09T08:52:01.155 INFO:teuthology.orchestra.run.smithi038.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-09T08:52:01.156 INFO:teuthology.orchestra.run.smithi038.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-09T08:52:01.157 INFO:teuthology.orchestra.run.smithi038.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-09T08:52:01.158 INFO:teuthology.orchestra.run.smithi038.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-09T08:52:01.159 INFO:teuthology.orchestra.run.smithi038.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-09T08:52:01.213 INFO:teuthology.orchestra.run.smithi041.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-09T08:52:01.213 INFO:teuthology.orchestra.run.smithi041.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-09T08:52:01.228 INFO:teuthology.orchestra.run.smithi041.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-09T08:52:01.228 INFO:teuthology.orchestra.run.smithi041.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-09T08:52:01.229 INFO:teuthology.orchestra.run.smithi041.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-09T08:52:01.229 INFO:teuthology.orchestra.run.smithi041.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-09T08:52:01.230 INFO:teuthology.orchestra.run.smithi041.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-09T08:52:01.230 INFO:teuthology.orchestra.run.smithi041.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-09T08:52:01.230 INFO:teuthology.orchestra.run.smithi041.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-09T08:52:01.234 INFO:teuthology.orchestra.run.smithi038.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-09T08:52:01.236 INFO:teuthology.orchestra.run.smithi038.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-09T08:52:01.304 INFO:teuthology.orchestra.run.smithi041.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-09T08:52:01.306 INFO:teuthology.orchestra.run.smithi041.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-09T08:52:01.315 INFO:teuthology.orchestra.run.smithi038.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-09T08:52:01.316 INFO:teuthology.orchestra.run.smithi038.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-09T08:52:01.322 INFO:teuthology.orchestra.run.smithi038.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-09T08:52:01.323 INFO:teuthology.orchestra.run.smithi038.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-09T08:52:01.324 INFO:teuthology.orchestra.run.smithi038.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-09T08:52:01.326 INFO:teuthology.orchestra.run.smithi038.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-09T08:52:01.327 INFO:teuthology.orchestra.run.smithi038.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-09T08:52:01.328 INFO:teuthology.orchestra.run.smithi038.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-09T08:52:01.384 INFO:teuthology.orchestra.run.smithi041.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-09T08:52:01.384 INFO:teuthology.orchestra.run.smithi041.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-09T08:52:01.390 INFO:teuthology.orchestra.run.smithi041.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-09T08:52:01.391 INFO:teuthology.orchestra.run.smithi041.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-09T08:52:01.391 INFO:teuthology.orchestra.run.smithi041.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-09T08:52:01.395 INFO:teuthology.orchestra.run.smithi041.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-09T08:52:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-09T08:52:01.397 INFO:teuthology.orchestra.run.smithi041.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-09T08:52:01.397 INFO:teuthology.orchestra.run.smithi041.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-09T08:52:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-09T08:52:01.462 INFO:teuthology.orchestra.run.smithi041.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-09T08:52:01.468 INFO:teuthology.orchestra.run.smithi041.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-09T08:52:01.477 INFO:teuthology.orchestra.run.smithi038.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-09T08:52:01.485 INFO:teuthology.orchestra.run.smithi038.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-09T08:52:01.485 INFO:teuthology.orchestra.run.smithi038.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-09T08:52:01.486 INFO:teuthology.orchestra.run.smithi038.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-09T08:52:01.487 INFO:teuthology.orchestra.run.smithi038.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-09T08:52:01.488 INFO:teuthology.orchestra.run.smithi038.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-09T08:52:01.491 INFO:teuthology.orchestra.run.smithi038.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-09T08:52:01.491 INFO:teuthology.orchestra.run.smithi038.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-09T08:52:01.542 INFO:teuthology.orchestra.run.smithi041.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-09T08:52:01.548 INFO:teuthology.orchestra.run.smithi041.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-09T08:52:01.550 INFO:teuthology.orchestra.run.smithi041.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-09T08:52:01.550 INFO:teuthology.orchestra.run.smithi041.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-09T08:52:01.551 INFO:teuthology.orchestra.run.smithi041.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-09T08:52:01.552 INFO:teuthology.orchestra.run.smithi041.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-09T08:52:01.554 INFO:teuthology.orchestra.run.smithi041.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-09T08:52:01.554 INFO:teuthology.orchestra.run.smithi041.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-09T08:52:01.557 INFO:teuthology.orchestra.run.smithi038.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-09T08:52:01.558 INFO:teuthology.orchestra.run.smithi038.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-09T08:52:01.621 INFO:teuthology.orchestra.run.smithi041.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-09T08:52:01.622 INFO:teuthology.orchestra.run.smithi041.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-09T08:52:01.623 INFO:teuthology.orchestra.run.smithi038.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1bionic [5,646 kB] 2021-03-09T08:52:01.639 INFO:teuthology.orchestra.run.smithi038.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-09T08:52:01.661 INFO:teuthology.orchestra.run.smithi038.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-09T08:52:01.663 INFO:teuthology.orchestra.run.smithi038.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-09T08:52:01.663 INFO:teuthology.orchestra.run.smithi038.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-09T08:52:01.700 INFO:teuthology.orchestra.run.smithi041.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-09T08:52:01.723 INFO:teuthology.orchestra.run.smithi041.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-09T08:52:01.724 INFO:teuthology.orchestra.run.smithi041.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-09T08:52:01.724 INFO:teuthology.orchestra.run.smithi041.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-09T08:52:01.803 INFO:teuthology.orchestra.run.smithi038.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-09T08:52:01.804 INFO:teuthology.orchestra.run.smithi038.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-09T08:52:01.806 INFO:teuthology.orchestra.run.smithi038.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-09T08:52:01.815 INFO:teuthology.orchestra.run.smithi038.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-09T08:52:01.815 INFO:teuthology.orchestra.run.smithi038.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-09T08:52:01.816 INFO:teuthology.orchestra.run.smithi038.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-09T08:52:01.817 INFO:teuthology.orchestra.run.smithi038.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-09T08:52:01.818 INFO:teuthology.orchestra.run.smithi038.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-09T08:52:01.819 INFO:teuthology.orchestra.run.smithi038.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-09T08:52:01.850 INFO:teuthology.orchestra.run.smithi041.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-09T08:52:01.851 INFO:teuthology.orchestra.run.smithi041.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-09T08:52:01.852 INFO:teuthology.orchestra.run.smithi041.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-09T08:52:01.858 INFO:teuthology.orchestra.run.smithi041.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-09T08:52:01.859 INFO:teuthology.orchestra.run.smithi041.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-09T08:52:01.860 INFO:teuthology.orchestra.run.smithi041.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-09T08:52:01.860 INFO:teuthology.orchestra.run.smithi041.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-09T08:52:01.862 INFO:teuthology.orchestra.run.smithi041.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-09T08:52:01.862 INFO:teuthology.orchestra.run.smithi041.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-09T08:52:01.945 INFO:teuthology.orchestra.run.smithi038.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1bionic [184 kB] 2021-03-09T08:52:01.951 INFO:teuthology.orchestra.run.smithi038.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1bionic [1,388 kB] 2021-03-09T08:52:02.025 INFO:teuthology.orchestra.run.smithi038.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1bionic [6,670 kB] 2021-03-09T08:52:02.389 INFO:teuthology.orchestra.run.smithi038.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1bionic [24.9 MB] 2021-03-09T08:52:02.574 INFO:teuthology.orchestra.run.smithi041.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1bionic [5,646 kB] 2021-03-09T08:52:03.028 INFO:teuthology.orchestra.run.smithi041.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1bionic [184 kB] 2021-03-09T08:52:03.034 INFO:teuthology.orchestra.run.smithi041.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1bionic [1,388 kB] 2021-03-09T08:52:03.172 INFO:teuthology.orchestra.run.smithi041.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1bionic [6,670 kB] 2021-03-09T08:52:03.719 INFO:teuthology.orchestra.run.smithi041.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1bionic [24.9 MB] 2021-03-09T08:52:03.761 INFO:teuthology.orchestra.run.smithi038.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1bionic [3,812 B] 2021-03-09T08:52:03.761 INFO:teuthology.orchestra.run.smithi038.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1bionic [771 kB] 2021-03-09T08:52:03.784 INFO:teuthology.orchestra.run.smithi038.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1bionic [2,153 kB] 2021-03-09T08:52:03.919 INFO:teuthology.orchestra.run.smithi038.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1bionic [63.1 kB] 2021-03-09T08:52:03.920 INFO:teuthology.orchestra.run.smithi038.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1bionic [68.9 kB] 2021-03-09T08:52:03.923 INFO:teuthology.orchestra.run.smithi038.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1bionic [2,333 kB] 2021-03-09T08:52:04.064 INFO:teuthology.orchestra.run.smithi038.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1bionic [7,881 kB] 2021-03-09T08:52:04.486 INFO:teuthology.orchestra.run.smithi038.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1bionic [3,928 B] 2021-03-09T08:52:04.486 INFO:teuthology.orchestra.run.smithi038.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1bionic [22.5 kB] 2021-03-09T08:52:04.486 INFO:teuthology.orchestra.run.smithi038.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1bionic [45.2 MB] 2021-03-09T08:52:05.587 INFO:teuthology.orchestra.run.smithi041.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1bionic [3,812 B] 2021-03-09T08:52:05.588 INFO:teuthology.orchestra.run.smithi041.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1bionic [771 kB] 2021-03-09T08:52:05.611 INFO:teuthology.orchestra.run.smithi041.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1bionic [2,153 kB] 2021-03-09T08:52:05.753 INFO:teuthology.orchestra.run.smithi041.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1bionic [63.1 kB] 2021-03-09T08:52:05.787 INFO:teuthology.orchestra.run.smithi041.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1bionic [68.9 kB] 2021-03-09T08:52:05.794 INFO:teuthology.orchestra.run.smithi041.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1bionic [2,333 kB] 2021-03-09T08:52:05.945 INFO:teuthology.orchestra.run.smithi041.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1bionic [7,881 kB] 2021-03-09T08:52:06.468 INFO:teuthology.orchestra.run.smithi041.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1bionic [3,928 B] 2021-03-09T08:52:06.502 INFO:teuthology.orchestra.run.smithi041.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1bionic [22.5 kB] 2021-03-09T08:52:06.503 INFO:teuthology.orchestra.run.smithi041.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1bionic [45.2 MB] 2021-03-09T08:52:07.284 INFO:teuthology.orchestra.run.smithi038.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1bionic [19.9 kB] 2021-03-09T08:52:07.284 INFO:teuthology.orchestra.run.smithi038.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1bionic [10.2 MB] 2021-03-09T08:52:07.885 INFO:teuthology.orchestra.run.smithi038.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1bionic [77.7 kB] 2021-03-09T08:52:08.609 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 166 MB in 8s (20.1 MB/s) 2021-03-09T08:52:08.729 INFO:teuthology.orchestra.run.smithi041.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1bionic [19.9 kB] 2021-03-09T08:52:08.729 INFO:teuthology.orchestra.run.smithi041.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1bionic [10.2 MB] 2021-03-09T08:52:08.730 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-09T08:52:08.766 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-09T08:52:08.769 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-09T08:52:08.804 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T08:52:09.022 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libjaeger. 2021-03-09T08:52:09.033 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../001-libjaeger_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:09.061 INFO:teuthology.orchestra.run.smithi041.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1bionic [77.7 kB] 2021-03-09T08:52:09.072 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:09.400 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T08:52:09.411 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-09T08:52:09.501 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T08:52:09.790 INFO:teuthology.orchestra.run.smithi041.stdout:Fetched 166 MB in 9s (17.7 MB/s) 2021-03-09T08:52:09.841 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../003-librbd1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:09.935 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-09T08:52:09.962 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T08:52:09.973 INFO:teuthology.orchestra.run.smithi041.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-09T08:52:09.976 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-09T08:52:10.017 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T08:52:10.244 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libjaeger. 2021-03-09T08:52:10.256 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../001-libjaeger_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:10.286 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:10.597 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../004-librados2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:10.717 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T08:52:10.882 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T08:52:10.892 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-09T08:52:10.972 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T08:52:11.303 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../003-librbd1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:11.426 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T08:52:11.487 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T08:52:11.499 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:11.537 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:11.881 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rados. 2021-03-09T08:52:11.893 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../006-python3-rados_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:11.931 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:12.091 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../004-librados2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:12.213 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T08:52:12.216 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T08:52:12.229 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:12.267 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:12.536 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T08:52:12.546 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:12.586 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:12.842 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T08:52:12.854 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:12.892 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:12.922 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T08:52:12.933 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:12.972 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:13.299 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T08:52:13.312 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-03-09T08:52:13.336 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-rados. 2021-03-09T08:52:13.348 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../006-python3-rados_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:13.349 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-09T08:52:13.385 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:13.568 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T08:52:13.580 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:13.610 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:13.713 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T08:52:13.726 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:13.763 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:13.996 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T08:52:14.008 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-09T08:52:14.015 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T08:52:14.028 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:14.046 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T08:52:14.065 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:14.368 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T08:52:14.380 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:14.407 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T08:52:14.417 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:14.420 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-09T08:52:14.457 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T08:52:14.728 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T08:52:14.741 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-03-09T08:52:14.777 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librgw2. 2021-03-09T08:52:14.787 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-09T08:52:14.788 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:14.818 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:15.032 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T08:52:15.043 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:15.081 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:15.362 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T08:52:15.374 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:15.412 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:15.459 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T08:52:15.469 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-09T08:52:15.508 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T08:52:15.756 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-09T08:52:15.768 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-09T08:52:15.806 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T08:52:15.811 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T08:52:15.824 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-09T08:52:15.861 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T08:52:16.084 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-09T08:52:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-09T08:52:16.143 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T08:52:16.206 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package librgw2. 2021-03-09T08:52:16.218 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:16.255 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:16.462 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liboath0. 2021-03-09T08:52:16.474 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-03-09T08:52:16.503 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-09T08:52:16.808 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T08:52:16.814 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T08:52:16.818 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:16.825 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:16.857 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:16.864 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:17.169 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-09T08:52:17.181 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-09T08:52:17.210 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T08:52:17.334 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-common. 2021-03-09T08:52:17.345 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../020-ceph-common_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:17.383 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:17.513 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-09T08:52:17.525 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-09T08:52:17.563 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T08:52:17.957 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package liboath0. 2021-03-09T08:52:17.969 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-03-09T08:52:18.007 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-09T08:52:18.318 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T08:52:18.329 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:18.368 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:18.754 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-common. 2021-03-09T08:52:18.766 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../020-ceph-common_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:18.804 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:19.305 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-base. 2021-03-09T08:52:19.317 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../021-ceph-base_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:19.365 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:20.435 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-dateutil. 2021-03-09T08:52:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-03-09T08:52:20.477 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-09T08:52:20.734 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-base. 2021-03-09T08:52:20.746 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../021-ceph-base_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:20.747 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-09T08:52:20.758 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-09T08:52:20.788 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-09T08:52:20.793 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:21.040 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-09T08:52:21.051 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-09T08:52:21.090 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-09T08:52:21.434 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T08:52:21.447 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-09T08:52:21.484 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T08:52:21.705 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-dateutil. 2021-03-09T08:52:21.717 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-03-09T08:52:21.755 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-09T08:52:21.854 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T08:52:21.866 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-09T08:52:21.904 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T08:52:22.066 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-09T08:52:22.078 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-09T08:52:22.116 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-09T08:52:22.173 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T08:52:22.185 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:22.223 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:22.385 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-09T08:52:22.397 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-09T08:52:22.435 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-09T08:52:22.636 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T08:52:22.648 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-09T08:52:22.686 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-09T08:52:22.746 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T08:52:22.758 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-09T08:52:22.796 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T08:52:22.922 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T08:52:22.934 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-09T08:52:22.972 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-09T08:52:23.157 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T08:52:23.170 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-09T08:52:23.207 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T08:52:23.342 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T08:52:23.355 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-09T08:52:23.383 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-09T08:52:23.460 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T08:52:23.473 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:23.510 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:23.695 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T08:52:23.706 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-03-09T08:52:23.746 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-09T08:52:24.015 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T08:52:24.026 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-09T08:52:24.064 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-09T08:52:24.081 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T08:52:24.092 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-09T08:52:24.131 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-09T08:52:24.366 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T08:52:24.379 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-09T08:52:24.408 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-09T08:52:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-mako. 2021-03-09T08:52:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-09T08:52:24.491 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-09T08:52:24.769 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T08:52:24.782 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-09T08:52:24.903 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-09T08:52:24.903 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T08:52:24.904 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-09T08:52:24.904 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-09T08:52:25.072 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T08:52:25.087 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-03-09T08:52:25.122 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-09T08:52:25.138 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T08:52:25.151 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-09T08:52:25.188 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-09T08:52:25.425 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T08:52:25.436 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-09T08:52:25.475 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-09T08:52:25.516 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webob. 2021-03-09T08:52:25.528 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-09T08:52:25.566 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T08:52:25.777 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-mako. 2021-03-09T08:52:25.789 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-09T08:52:25.827 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-09T08:52:25.886 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T08:52:25.899 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-03-09T08:52:25.935 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-09T08:52:26.147 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T08:52:26.158 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-09T08:52:26.197 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-09T08:52:26.263 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T08:52:26.275 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-03-09T08:52:26.314 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-09T08:52:26.491 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T08:52:26.504 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-09T08:52:26.532 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-09T08:52:26.641 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T08:52:26.654 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-03-09T08:52:26.691 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-09T08:52:26.818 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-webob. 2021-03-09T08:52:26.830 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-09T08:52:26.870 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T08:52:26.985 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-paste. 2021-03-09T08:52:26.997 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-09T08:52:27.043 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T08:52:27.179 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T08:52:27.191 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-03-09T08:52:27.229 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-09T08:52:27.338 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T08:52:27.351 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-09T08:52:27.388 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T08:52:27.540 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T08:52:27.553 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-03-09T08:52:27.590 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-09T08:52:27.690 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T08:52:27.703 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-09T08:52:27.740 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-09T08:52:27.909 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T08:52:27.921 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-03-09T08:52:27.959 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-09T08:52:28.035 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T08:52:28.047 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-09T08:52:28.085 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T08:52:28.279 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-paste. 2021-03-09T08:52:28.292 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-09T08:52:28.329 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T08:52:28.404 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T08:52:28.417 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-03-09T08:52:28.454 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-09T08:52:28.647 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T08:52:28.659 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-09T08:52:28.697 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T08:52:28.767 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T08:52:28.778 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-09T08:52:28.817 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T08:52:29.000 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T08:52:29.013 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-09T08:52:29.075 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-09T08:52:29.200 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T08:52:29.212 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:29.242 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:29.369 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T08:52:29.382 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-09T08:52:29.420 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T08:52:29.712 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T08:52:29.724 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:29.755 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T08:52:29.763 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:29.767 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-03-09T08:52:29.807 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-09T08:52:30.134 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T08:52:30.147 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-09T08:52:30.184 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T08:52:30.524 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T08:52:30.537 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:30.574 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:30.740 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T08:52:30.753 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:30.790 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:31.710 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T08:52:31.723 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:31.762 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:32.722 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T08:52:32.734 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:32.780 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:33.145 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph. 2021-03-09T08:52:33.157 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../049-ceph_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:33.195 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:33.540 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T08:52:33.551 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:33.590 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:33.960 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T08:52:33.972 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:34.009 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:34.504 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package cephadm. 2021-03-09T08:52:34.516 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../052-cephadm_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:34.554 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:34.890 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T08:52:34.901 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph. 2021-03-09T08:52:34.902 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-09T08:52:34.913 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../049-ceph_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:34.940 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T08:52:34.952 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:35.237 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T08:52:35.243 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T08:52:35.249 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:35.254 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:35.285 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:35.287 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:35.630 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T08:52:35.641 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-03-09T08:52:35.679 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-09T08:52:35.691 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T08:52:35.702 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:35.741 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:35.982 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-routes. 2021-03-09T08:52:35.994 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-03-09T08:52:36.032 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-09T08:52:36.276 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package cephadm. 2021-03-09T08:52:36.288 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../052-cephadm_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:36.326 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:36.369 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T08:52:36.380 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:36.419 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:36.587 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T08:52:36.601 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-09T08:52:36.629 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T08:52:36.899 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T08:52:36.910 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:36.948 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:37.235 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T08:52:37.248 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-03-09T08:52:37.277 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-09T08:52:37.401 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-numpy. 2021-03-09T08:52:37.415 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-09T08:52:37.451 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T08:52:37.579 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-routes. 2021-03-09T08:52:37.591 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-03-09T08:52:37.629 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-09T08:52:37.940 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T08:52:37.953 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:37.990 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:38.063 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T08:52:38.077 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-09T08:52:38.122 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-09T08:52:38.433 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T08:52:38.444 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-09T08:52:38.483 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T08:52:38.916 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-numpy. 2021-03-09T08:52:38.928 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-09T08:52:38.972 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T08:52:39.541 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T08:52:39.554 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-09T08:52:39.591 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-09T08:52:39.666 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T08:52:39.680 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-09T08:52:39.724 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-09T08:52:39.894 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T08:52:39.905 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-09T08:52:39.944 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T08:52:40.211 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T08:52:40.222 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-09T08:52:40.261 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-09T08:52:40.615 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T08:52:40.625 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-09T08:52:40.664 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-09T08:52:41.138 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T08:52:41.191 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-09T08:52:41.236 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T08:52:41.248 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:41.286 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:41.360 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-09T08:52:41.784 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T08:52:41.797 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-09T08:52:41.839 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-09T08:52:42.192 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T08:52:42.204 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-09T08:52:42.242 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-09T08:52:42.648 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-monotonic. 2021-03-09T08:52:42.660 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-09T08:52:42.690 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-09T08:52:42.804 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T08:52:42.817 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:42.853 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:43.018 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-fasteners. 2021-03-09T08:52:43.030 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-09T08:52:43.068 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-09T08:52:43.371 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-httplib2. 2021-03-09T08:52:43.383 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-09T08:52:43.421 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T08:52:43.715 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T08:52:43.729 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-09T08:52:43.757 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-09T08:52:43.990 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-monotonic. 2021-03-09T08:52:44.003 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-09T08:52:44.039 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-09T08:52:44.052 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T08:52:44.064 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-09T08:52:44.102 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T08:52:44.334 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-fasteners. 2021-03-09T08:52:44.346 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-09T08:52:44.384 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-09T08:52:44.438 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T08:52:44.451 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-09T08:52:44.488 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-09T08:52:44.670 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-httplib2. 2021-03-09T08:52:44.684 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-09T08:52:44.720 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T08:52:44.800 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-09T08:52:44.813 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-09T08:52:44.850 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T08:52:45.006 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T08:52:45.019 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-09T08:52:45.056 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-09T08:52:45.288 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-09T08:52:45.299 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-09T08:52:45.337 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-09T08:52:45.376 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T08:52:45.389 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-09T08:52:45.426 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T08:52:45.699 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-setuptools. 2021-03-09T08:52:45.713 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-09T08:52:45.737 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T08:52:45.749 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-09T08:52:45.749 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-09T08:52:45.787 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-09T08:52:46.099 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-09T08:52:46.111 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-09T08:52:46.119 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T08:52:46.133 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-09T08:52:46.149 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T08:52:46.177 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T08:52:46.481 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T08:52:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-09T08:52:46.531 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T08:52:46.604 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-09T08:52:46.618 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-09T08:52:46.654 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-09T08:52:46.960 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T08:52:46.973 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:46.981 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-setuptools. 2021-03-09T08:52:46.996 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-09T08:52:47.002 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:47.032 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-09T08:52:47.307 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T08:52:47.326 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-09T08:52:47.355 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-09T08:52:47.426 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T08:52:47.438 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-09T08:52:47.496 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T08:52:47.716 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T08:52:47.728 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-09T08:52:47.769 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T08:52:47.812 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T08:52:47.824 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-09T08:52:47.862 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T08:52:48.069 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T08:52:48.082 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:48.111 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:48.292 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T08:52:48.306 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:48.350 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:48.439 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-09T08:52:48.451 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-09T08:52:48.574 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-09T08:52:48.669 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T08:52:48.681 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-09T08:52:48.718 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-09T08:52:48.927 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T08:52:48.940 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-09T08:52:48.977 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T08:52:49.022 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T08:52:49.034 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-09T08:52:49.075 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T08:52:49.197 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package jq. 2021-03-09T08:52:49.210 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-09T08:52:49.255 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-09T08:52:49.367 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T08:52:49.379 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T08:52:49.417 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:49.525 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package socat. 2021-03-09T08:52:49.538 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-09T08:52:49.616 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-09T08:52:49.761 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-09T08:52:49.774 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-09T08:52:49.896 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-09T08:52:50.003 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T08:52:50.015 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-09T08:52:50.053 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T08:52:50.233 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T08:52:50.246 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-09T08:52:50.283 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T08:52:50.331 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T08:52:50.345 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-09T08:52:50.389 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-09T08:52:50.544 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package jq. 2021-03-09T08:52:50.555 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-09T08:52:50.594 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-09T08:52:50.736 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-test. 2021-03-09T08:52:50.747 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../086-ceph-test_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:50.785 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:50.855 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package socat. 2021-03-09T08:52:50.869 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-09T08:52:50.913 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-09T08:52:51.258 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T08:52:51.272 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-09T08:52:51.308 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T08:52:51.578 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T08:52:51.597 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-09T08:52:51.636 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-09T08:52:51.983 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package ceph-test. 2021-03-09T08:52:51.994 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../086-ceph-test_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:52.032 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:54.181 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T08:52:54.194 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:54.239 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:54.525 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T08:52:54.539 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-09T08:52:54.584 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T08:52:54.922 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-attr. 2021-03-09T08:52:54.935 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-09T08:52:54.988 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-09T08:52:55.233 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-funcsigs. 2021-03-09T08:52:55.246 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-09T08:52:55.275 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-09T08:52:55.578 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pluggy. 2021-03-09T08:52:55.591 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-09T08:52:55.628 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-09T08:52:55.685 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T08:52:55.698 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:52:55.735 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:52:55.930 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-py. 2021-03-09T08:52:55.943 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-09T08:52:55.980 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-09T08:52:56.013 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T08:52:56.026 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-09T08:52:56.063 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T08:52:56.359 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pytest. 2021-03-09T08:52:56.371 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-09T08:52:56.384 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python-attr. 2021-03-09T08:52:56.396 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-09T08:52:56.409 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-09T08:52:56.426 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-09T08:52:56.746 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-attr. 2021-03-09T08:52:56.754 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python-funcsigs. 2021-03-09T08:52:56.757 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-09T08:52:56.767 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-09T08:52:56.795 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-09T08:52:56.812 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-09T08:52:57.106 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python-pluggy. 2021-03-09T08:52:57.119 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-09T08:52:57.156 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-09T08:52:57.174 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cycler. 2021-03-09T08:52:57.186 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-09T08:52:57.215 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-09T08:52:57.468 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python-py. 2021-03-09T08:52:57.479 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-09T08:52:57.502 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T08:52:57.515 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-09T08:52:57.517 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-09T08:52:57.552 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-09T08:52:57.846 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T08:52:57.854 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python-pytest. 2021-03-09T08:52:57.860 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-09T08:52:57.866 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-09T08:52:57.904 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-09T08:52:57.905 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-09T08:52:58.225 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T08:52:58.239 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-09T08:52:58.241 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-attr. 2021-03-09T08:52:58.253 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-09T08:52:58.283 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T08:52:58.290 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-09T08:52:58.544 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-cycler. 2021-03-09T08:52:58.557 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-09T08:52:58.593 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-09T08:52:58.720 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-09T08:52:58.732 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-09T08:52:58.769 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T08:52:58.896 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T08:52:58.909 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-09T08:52:58.946 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-09T08:52:59.106 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tz. 2021-03-09T08:52:59.119 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-09T08:52:59.155 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-09T08:52:59.241 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T08:52:59.254 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-09T08:52:59.299 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-09T08:52:59.442 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-09T08:52:59.455 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-09T08:52:59.492 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T08:52:59.635 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T08:52:59.649 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-09T08:52:59.685 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T08:53:00.097 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-09T08:53:00.111 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-09T08:53:00.155 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T08:53:00.181 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-olefile. 2021-03-09T08:53:00.374 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-09T08:53:00.376 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-09T08:53:00.442 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-tz. 2021-03-09T08:53:00.455 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-09T08:53:00.494 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-09T08:53:00.576 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T08:53:00.588 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-09T08:53:00.626 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-09T08:53:00.803 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-09T08:53:00.815 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-09T08:53:00.852 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T08:53:00.971 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-09T08:53:00.983 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-09T08:53:01.021 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T08:53:01.383 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T08:53:01.395 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-09T08:53:01.432 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-09T08:53:01.508 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-olefile. 2021-03-09T08:53:01.522 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-09T08:53:01.566 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-09T08:53:01.752 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-py. 2021-03-09T08:53:01.765 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-09T08:53:01.810 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-09T08:53:01.844 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T08:53:01.859 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-09T08:53:01.894 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-09T08:53:02.114 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T08:53:02.128 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-09T08:53:02.172 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-09T08:53:02.264 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-09T08:53:02.276 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-09T08:53:02.314 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T08:53:02.484 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T08:53:02.496 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-09T08:53:02.533 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-09T08:53:02.668 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T08:53:02.679 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-09T08:53:02.709 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-09T08:53:02.878 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T08:53:02.891 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-09T08:53:02.928 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-09T08:53:02.995 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-py. 2021-03-09T08:53:03.009 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-09T08:53:03.045 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-09T08:53:03.281 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-09T08:53:03.295 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-09T08:53:03.324 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T08:53:03.331 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T08:53:03.336 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-09T08:53:03.365 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-09T08:53:03.676 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T08:53:03.689 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-09T08:53:03.726 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-09T08:53:03.786 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package radosgw. 2021-03-09T08:53:03.799 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../111-radosgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:53:03.835 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:04.188 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T08:53:04.201 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-09T08:53:04.242 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-09T08:53:04.574 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-09T08:53:04.586 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-09T08:53:04.624 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T08:53:04.891 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T08:53:04.903 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:53:04.939 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:05.053 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package radosgw. 2021-03-09T08:53:05.067 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../111-radosgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:53:05.113 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:05.247 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-09T08:53:05.611 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T08:53:05.979 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T08:53:06.130 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T08:53:06.139 INFO:teuthology.orchestra.run.smithi041.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T08:53:06.152 INFO:teuthology.orchestra.run.smithi041.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T08:53:06.181 INFO:teuthology.orchestra.run.smithi041.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:06.480 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-09T08:53:06.760 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T08:53:06.803 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T08:53:07.162 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-09T08:53:07.170 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T08:53:07.272 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T08:53:07.420 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-09T08:53:07.713 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-09T08:53:07.874 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T08:53:07.906 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-09T08:53:08.032 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-09T08:53:08.259 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-09T08:53:08.404 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-09T08:53:08.543 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-09T08:53:08.780 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-09T08:53:08.844 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-09T08:53:09.037 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-09T08:53:09.070 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-09T08:53:09.138 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-09T08:53:09.338 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:09.464 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-09T08:53:09.639 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-09T08:53:09.805 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-09T08:53:09.965 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-09T08:53:10.091 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T08:53:10.109 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-09T08:53:10.410 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:10.711 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-09T08:53:11.012 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-09T08:53:11.044 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T08:53:11.113 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T08:53:11.371 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-09T08:53:11.655 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-py (1.5.2-1) ... 2021-03-09T08:53:11.890 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-09T08:53:12.084 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T08:53:12.224 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-09T08:53:12.314 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-09T08:53:12.393 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-09T08:53:12.559 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-09T08:53:12.685 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python-py (1.5.2-1) ... 2021-03-09T08:53:12.920 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-09T08:53:13.002 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-09T08:53:13.187 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-09T08:53:13.271 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-09T08:53:13.343 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-09T08:53:13.471 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T08:53:13.582 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-09T08:53:13.682 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-09T08:53:13.782 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-09T08:53:13.874 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-09T08:53:14.056 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-09T08:53:14.167 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T08:53:14.416 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-09T08:53:14.501 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-09T08:53:14.751 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-09T08:53:14.794 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-09T08:53:15.060 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:15.170 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-09T08:53:15.480 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-09T08:53:15.702 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user cephadm....done 2021-03-09T08:53:15.748 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:15.841 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-09T08:53:16.135 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T08:53:16.260 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T08:53:16.404 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-09T08:53:16.415 INFO:teuthology.orchestra.run.smithi041.stdout:Adding system user cephadm....done 2021-03-09T08:53:16.579 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-09T08:53:16.771 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-09T08:53:16.872 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T08:53:16.973 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T08:53:17.055 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T08:53:17.090 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-09T08:53:17.156 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-09T08:53:17.433 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-09T08:53:17.458 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:17.734 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T08:53:17.759 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-09T08:53:17.863 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-09T08:53:18.102 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T08:53:18.187 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:18.228 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T08:53:18.505 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-09T08:53:18.554 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-09T08:53:18.780 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T08:53:18.882 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T08:53:18.914 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T08:53:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T08:53:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-09T08:53:19.216 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-09T08:53:19.249 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-09T08:53:19.542 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T08:53:19.576 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T08:53:19.666 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T08:53:19.698 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-09T08:53:19.843 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T08:53:19.936 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-09T08:53:20.186 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-09T08:53:20.246 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T08:53:20.512 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-09T08:53:20.521 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T08:53:20.705 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:20.815 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T08:53:20.856 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-09T08:53:21.143 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-09T08:53:21.207 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T08:53:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T08:53:21.341 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:21.476 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T08:53:21.547 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-09T08:53:21.727 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T08:53:21.879 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T08:53:21.894 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T08:53:21.984 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T08:53:22.129 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T08:53:22.237 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-09T08:53:22.255 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-09T08:53:22.388 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T08:53:22.490 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T08:53:22.623 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T08:53:22.724 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-09T08:53:22.732 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T08:53:22.858 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-09T08:53:23.025 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-09T08:53:23.252 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T08:53:23.377 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-09T08:53:23.385 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T08:53:23.686 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-09T08:53:23.687 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-09T08:53:23.980 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T08:53:23.987 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-09T08:53:24.256 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-09T08:53:24.355 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-09T08:53:24.532 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-09T08:53:24.656 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-09T08:53:24.883 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-09T08:53:24.932 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-09T08:53:25.201 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-09T08:53:25.233 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T08:53:25.502 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-09T08:53:25.602 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-09T08:53:25.828 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T08:53:25.920 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-09T08:53:26.045 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T08:53:26.196 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-09T08:53:26.489 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-09T08:53:26.590 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T08:53:27.259 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T08:53:27.594 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-09T08:53:27.763 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T08:53:27.928 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-09T08:53:28.048 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-09T08:53:28.063 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-09T08:53:28.357 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-09T08:53:28.447 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-09T08:53:28.483 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-09T08:53:28.682 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T08:53:28.784 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-09T08:53:28.851 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-09T08:53:28.889 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.010 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-09T08:53:29.128 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T08:53:29.253 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-09T08:53:29.337 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.379 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.462 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.480 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-09T08:53:29.588 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.714 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T08:53:29.765 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.865 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:29.983 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:30.031 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-09T08:53:30.117 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T08:53:30.316 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-09T08:53:30.443 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-09T08:53:30.710 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T08:53:30.769 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-09T08:53:31.172 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T08:53:31.587 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-09T08:53:32.025 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-09T08:53:32.443 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T08:53:32.736 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:32.862 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:32.871 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T08:53:32.988 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.122 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.164 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.264 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.346 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:53:33.346 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:53:33.365 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.466 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.615 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:33.681 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:53:33.682 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:53:33.716 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-09T08:53:33.789 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-09T08:53:33.830 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-09T08:53:33.964 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:34.085 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-09T08:53:34.145 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T08:53:34.176 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-09T08:53:34.215 INFO:teuthology.orchestra.run.smithi041.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-09T08:53:34.530 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T08:53:34.805 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-09T08:53:35.182 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.233 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.283 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-09T08:53:35.358 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.468 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.594 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.615 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.735 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.861 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:35.987 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:36.128 INFO:teuthology.orchestra.run.smithi038.stdout:Adding group ceph....done 2021-03-09T08:53:36.471 INFO:teuthology.orchestra.run.smithi041.stdout:Adding group ceph....done 2021-03-09T08:53:36.472 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user ceph....done 2021-03-09T08:53:36.698 INFO:teuthology.orchestra.run.smithi038.stdout:Setting system user ceph properties....done 2021-03-09T08:53:36.849 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T08:53:36.857 INFO:teuthology.orchestra.run.smithi041.stdout:Adding system user ceph....done 2021-03-09T08:53:37.049 INFO:teuthology.orchestra.run.smithi041.stdout:Setting system user ceph properties....done 2021-03-09T08:53:37.092 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T08:53:37.208 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T08:53:37.382 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:37.442 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T08:53:37.649 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:53:37.649 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:53:37.746 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:38.015 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:53:38.016 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:53:38.047 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:38.251 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:38.423 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:38.524 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:38.672 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T08:53:38.832 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T08:53:38.967 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:39.134 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:39.183 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:53:39.184 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:53:39.352 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:53:39.353 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:53:39.485 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:39.628 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:39.652 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:39.795 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.063 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:53:40.063 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:53:40.237 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:53:40.238 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:53:40.330 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.473 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.522 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.607 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.656 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.773 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:40.829 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:53:40.829 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:53:40.987 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:53:40.987 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:53:41.117 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:41.266 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:41.348 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:53:41.348 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:53:41.467 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:53:41.467 INFO:teuthology.orchestra.run.smithi041.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:53:41.610 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:41.737 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:41.759 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:41.879 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:41.893 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:42.030 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-09T08:53:42.036 INFO:teuthology.orchestra.run.smithi041.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T08:53:42.163 INFO:teuthology.orchestra.run.smithi041.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-09T08:53:42.231 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-09T08:53:42.363 INFO:teuthology.orchestra.run.smithi041.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-09T08:53:44.435 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-09T08:53:44.469 INFO:teuthology.orchestra.run.smithi038.stdout:ureadahead will be reprofiled on next reboot 2021-03-09T08:53:44.519 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-09T08:53:44.526 INFO:teuthology.orchestra.run.smithi041.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-09T08:53:44.560 INFO:teuthology.orchestra.run.smithi041.stdout:ureadahead will be reprofiled on next reboot 2021-03-09T08:53:44.609 INFO:teuthology.orchestra.run.smithi041.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-09T08:53:45.004 INFO:teuthology.orchestra.run.smithi038.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T08:53:45.007 DEBUG:teuthology.parallel:result is None 2021-03-09T08:53:45.073 INFO:teuthology.orchestra.run.smithi041.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T08:53:45.077 DEBUG:teuthology.parallel:result is None 2021-03-09T08:53:45.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo lsb_release -is 2021-03-09T08:53:45.133 INFO:teuthology.orchestra.run.smithi038.stdout:Ubuntu 2021-03-09T08:53:45.138 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:53:45.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:53:45.312 DEBUG:teuthology.orchestra.run.smithi038:> dpkg-query -W -f '${Version}' ceph 2021-03-09T08:53:45.338 INFO:teuthology.orchestra.run.smithi038.stdout:17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:53:45.339 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:53:45.340 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1bionic is installed. 2021-03-09T08:53:45.340 DEBUG:teuthology.orchestra.run.smithi041:> sudo lsb_release -is 2021-03-09T08:53:45.396 INFO:teuthology.orchestra.run.smithi041.stdout:Ubuntu 2021-03-09T08:53:45.402 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:53:45.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:53:45.549 DEBUG:teuthology.orchestra.run.smithi041:> dpkg-query -W -f '${Version}' ceph 2021-03-09T08:53:45.575 INFO:teuthology.orchestra.run.smithi041.stdout:17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:53:45.576 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T08:53:45.576 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1bionic is installed. 2021-03-09T08:53:45.577 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-09T08:53:45.582 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:45.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T08:53:45.599 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:45.600 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T08:53:45.632 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-09T08:53:45.633 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:45.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T08:53:45.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T08:53:45.713 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:45.713 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T08:53:45.728 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T08:53:45.784 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-09T08:53:45.785 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:45.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T08:53:45.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T08:53:45.857 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:45.858 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T08:53:45.911 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T08:53:45.968 INFO:teuthology.run_tasks:Running task ceph... 2021-03-09T08:53:46.064 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-09T08:53:46.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2021-03-09T08:53:46.068 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod 777 /var/log/ceph 2021-03-09T08:53:46.082 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-09T08:53:46.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T08:53:46.125 DEBUG:teuthology.orchestra.run.smithi041:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T08:53:46.139 INFO:tasks.ceph:Creating extra log directories... 2021-03-09T08:53:46.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T08:53:46.181 DEBUG:teuthology.orchestra.run.smithi041:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T08:53:46.202 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2021-03-09T08:53:46.204 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2021-03-09T08:53:46.205 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2021-03-09T08:53:46.205 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:46.206 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-03-09T08:53:46.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-03-09T08:53:46.319 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:46.319 DEBUG:teuthology.orchestra.run.smithi041:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-03-09T08:53:46.326 DEBUG:teuthology.orchestra.run.smithi041:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-03-09T08:53:46.411 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-03-09T08:53:46.412 INFO:tasks.ceph:config {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-09T08:53:46.412 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949099', 'branch': 'wip-kefu-testing-2021-03-09-1428', 'description': 'rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/workunits}', 'email': 'tchaikov@gmail.com', 'first_in_suite': False, 'job_id': '5949099', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 2, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '18.04', 'overrides': {'admin_socket': {'branch': 'wip-kefu-testing-2021-03-09-1428'}, 'ceph': {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-kefu-testing-2021-03-09-1428', 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 60, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'], ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-kefu-testing-2021-03-09-1428', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'targets': {'smithi038.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgPI/Pyly9/2O5ftdKiMCkImpqw/QxFdnpRiJZDbiKE46WaXotGOWYWDu7ZiAdcHvVcSl5Kxyt+JowaqPeWXO80bVcUPOm3joJosEoFH88Gin5vYKNXLr0+m4mNOs1KasAtHRzL7is0XiHWLAyRGWwSLea3y+R5D9R1w5f92vLd1NcYh5YH6dveJu5T+IGXh9ynBGj6iUsahOYxHRDkrSH8oQMOOC+KNKIwNF5h4onoiGYNcTBpZavSQIQ0opJH+SrkwY8eTKNLj83aFQl3uvxgXvOPOGGv36OvPgjENuWsIqo+vf7+5NUxmBFjhjqNe0fgQzVNuOAl7xM/FnALSKJ', 'smithi041.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxs8IXgVRubcQLWRZMcAk8SydE9roW7SnHmJFTZrNn9M/flcFLY9fV/utNfnSPvCHwxL1LBSZUplo58pB3rulUtRM0qV6yt1ptwVGm4hRL7QbCFnmAEBgwxfKWNp9LE9j86aU5orVOTVWHoPP25vW38K0LzTyQfhmWsZjPuwvO6QBjP/QwSCJkmaE+/76MX6oxCcjXgmiWXyT6loseF2+Dr34W907z3ocoepvG47ajWNvmeYbDKi5JmcTT1M2Xv60U0bBohcA1v3IwubKH9+jUlU/znWYysPutDGR04umbMmAlVxOhDGx3XBg2MGkUS7zd6vT0LFG68m31E6gQsEKN'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['mgr']}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-09_08:22:40', 'tube': 'smithi', 'user': 'kchai', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.25973'} 2021-03-09T08:53:46.413 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T08:53:46.418 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T08:53:46.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T08:53:46.469 DEBUG:teuthology.orchestra.run.smithi041:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T08:53:46.514 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:46.515 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T08:53:46.533 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T08:53:46.534 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T08:53:46.582 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T08:53:46.582 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.583 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-03-09T08:53:46.583 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.584 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2021-03-09 08:51:14.653775473 +0000 2021-03-09T08:53:46.584 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2021-03-09 08:51:14.529837476 +0000 2021-03-09T08:53:46.585 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2021-03-09 08:51:14.529837476 +0000 2021-03-09T08:53:46.585 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2021-03-09T08:53:46.591 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2021-03-09T08:53:46.592 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2021-03-09T08:53:46.592 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000380351 s, 1.3 MB/s 2021-03-09T08:53:46.596 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T08:53:46.646 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T08:53:46.646 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.646 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 350 Links: 1 2021-03-09T08:53:46.647 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.647 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2021-03-09 08:51:14.889657468 +0000 2021-03-09T08:53:46.647 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2021-03-09 08:51:14.773715469 +0000 2021-03-09T08:53:46.648 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2021-03-09 08:51:14.773715469 +0000 2021-03-09T08:53:46.648 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2021-03-09T08:53:46.655 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2021-03-09T08:53:46.655 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2021-03-09T08:53:46.656 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000398963 s, 1.3 MB/s 2021-03-09T08:53:46.660 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T08:53:46.709 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T08:53:46.710 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.710 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 356 Links: 1 2021-03-09T08:53:46.710 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.711 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2021-03-09 08:51:15.121541462 +0000 2021-03-09T08:53:46.711 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2021-03-09 08:51:15.009597464 +0000 2021-03-09T08:53:46.711 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2021-03-09 08:51:15.009597464 +0000 2021-03-09T08:53:46.712 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2021-03-09T08:53:46.719 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2021-03-09T08:53:46.720 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2021-03-09T08:53:46.720 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000442577 s, 1.2 MB/s 2021-03-09T08:53:46.724 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T08:53:46.773 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T08:53:46.774 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.774 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 362 Links: 1 2021-03-09T08:53:46.774 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.775 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2021-03-09 08:51:15.385409457 +0000 2021-03-09T08:53:46.775 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2021-03-09 08:51:15.269467460 +0000 2021-03-09T08:53:46.775 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2021-03-09 08:51:15.269467460 +0000 2021-03-09T08:53:46.775 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2021-03-09T08:53:46.784 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2021-03-09T08:53:46.784 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2021-03-09T08:53:46.784 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00047112 s, 1.1 MB/s 2021-03-09T08:53:46.789 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2021-03-09T08:53:46.789 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:46.790 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T08:53:46.797 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T08:53:46.797 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T08:53:46.848 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T08:53:46.848 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.848 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 339 Links: 1 2021-03-09T08:53:46.849 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.849 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-03-09 08:51:14.514962907 +0000 2021-03-09T08:53:46.849 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-03-09 08:51:14.406908905 +0000 2021-03-09T08:53:46.849 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-03-09 08:51:14.406908905 +0000 2021-03-09T08:53:46.850 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-03-09T08:53:46.858 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-03-09T08:53:46.858 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-03-09T08:53:46.859 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000381811 s, 1.3 MB/s 2021-03-09T08:53:46.863 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T08:53:46.914 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T08:53:46.914 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.915 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-03-09T08:53:46.915 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.916 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-03-09 08:51:14.771090914 +0000 2021-03-09T08:53:46.916 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-03-09 08:51:14.663036911 +0000 2021-03-09T08:53:46.916 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-03-09 08:51:14.663036911 +0000 2021-03-09T08:53:46.917 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-03-09T08:53:46.923 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-03-09T08:53:46.924 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-03-09T08:53:46.924 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000288375 s, 1.8 MB/s 2021-03-09T08:53:46.926 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T08:53:46.976 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T08:53:46.977 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:46.977 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 352 Links: 1 2021-03-09T08:53:46.977 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:46.977 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-03-09 08:51:15.015212921 +0000 2021-03-09T08:53:46.977 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-03-09 08:51:14.887148917 +0000 2021-03-09T08:53:46.978 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-03-09 08:51:14.887148917 +0000 2021-03-09T08:53:46.978 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-03-09T08:53:46.986 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-03-09T08:53:46.986 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-03-09T08:53:46.986 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000380451 s, 1.3 MB/s 2021-03-09T08:53:46.990 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T08:53:47.040 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T08:53:47.040 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:53:47.041 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 358 Links: 1 2021-03-09T08:53:47.041 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:53:47.041 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-03-09 08:51:15.267338927 +0000 2021-03-09T08:53:47.041 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-03-09 08:51:15.147278924 +0000 2021-03-09T08:53:47.041 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-03-09 08:51:15.147278924 +0000 2021-03-09T08:53:47.042 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-03-09T08:53:47.049 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-03-09T08:53:47.049 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-03-09T08:53:47.049 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000411214 s, 1.2 MB/s 2021-03-09T08:53:47.053 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T08:53:47.054 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi041.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2021-03-09T08:53:47.054 INFO:tasks.ceph:Generating config... 2021-03-09T08:53:47.056 INFO:tasks.ceph:[client] debug client = 20 2021-03-09T08:53:47.057 INFO:tasks.ceph:[client] debug mgrc = 20 2021-03-09T08:53:47.057 INFO:tasks.ceph:[client] debug ms = 1 2021-03-09T08:53:47.057 INFO:tasks.ceph:[global] mon election default strategy = 3 2021-03-09T08:53:47.058 INFO:tasks.ceph:[mds] debug mgrc = 20 2021-03-09T08:53:47.058 INFO:tasks.ceph:[mgr] debug client = 20 2021-03-09T08:53:47.058 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-09T08:53:47.059 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-09T08:53:47.059 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-09T08:53:47.059 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-09T08:53:47.060 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-09T08:53:47.060 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-03-09T08:53:47.060 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2021-03-09T08:53:47.060 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-03-09T08:53:47.061 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-03-09T08:53:47.061 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-03-09T08:53:47.061 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-03-09T08:53:47.062 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-09T08:53:47.062 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-09T08:53:47.062 INFO:tasks.ceph:[osd] debug mgrc = 20 2021-03-09T08:53:47.062 INFO:tasks.ceph:[osd] debug ms = 20 2021-03-09T08:53:47.063 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-09T08:53:47.063 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-03-09T08:53:47.063 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-03-09T08:53:47.064 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-03-09T08:53:47.064 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-03-09T08:53:47.064 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-03-09T08:53:47.064 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-03-09T08:53:47.065 INFO:tasks.ceph:Setting up mon.a... 2021-03-09T08:53:47.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-03-09T08:53:47.156 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.keyring 2021-03-09T08:53:47.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-03-09T08:53:47.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T08:53:47.266 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.38'), ('mon.c', '[v2:172.21.15.38:3301,v1:172.21.15.38:6790]'), ('mon.b', '172.21.15.41')] 2021-03-09T08:53:47.267 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.41', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug mgrc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds': {'debug mgrc': 20}} 2021-03-09T08:53:47.268 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:47.268 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T08:53:47.313 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --addv c '[v2:172.21.15.38:3301,v1:172.21.15.38:6790]' --add b 172.21.15.41 --print /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:47.447 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid 988e2677-23e9-442a-94d8-290838bc06dd 2021-03-09T08:53:47.447 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = octopus 2021-03-09T08:53:47.447 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2021-03-09T08:53:47.447 INFO:teuthology.orchestra.run.smithi038.stdout:fsid 988e2677-23e9-442a-94d8-290838bc06dd 2021-03-09T08:53:47.447 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2021-03-09T08:53:47.446664+0000 2021-03-09T08:53:47.448 INFO:teuthology.orchestra.run.smithi038.stdout:created 2021-03-09T08:53:47.446664+0000 2021-03-09T08:53:47.448 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 15 (octopus) 2021-03-09T08:53:47.448 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 3 2021-03-09T08:53:47.449 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2021-03-09T08:53:47.449 INFO:teuthology.orchestra.run.smithi038.stdout:1: [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] mon.b 2021-03-09T08:53:47.449 INFO:teuthology.orchestra.run.smithi038.stdout:2: [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] mon.c 2021-03-09T08:53:47.449 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-03-09T08:53:47.451 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T08:53:47.456 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 988e2677-23e9-442a-94d8-290838bc06dd... 2021-03-09T08:53:47.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T08:53:47.501 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T08:53:47.530 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2021-03-09T08:53:47.530 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2021-03-09T08:53:47.531 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T08:53:47.531 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2021-03-09T08:53:47.531 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.531 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2021-03-09T08:53:47.532 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.532 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2021-03-09T08:53:47.532 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.532 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2021-03-09T08:53:47.533 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2021-03-09T08:53:47.533 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.533 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2021-03-09T08:53:47.533 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2021-03-09T08:53:47.534 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.534 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T08:53:47.534 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2021-03-09T08:53:47.534 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.535 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T08:53:47.535 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T08:53:47.535 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.535 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2021-03-09T08:53:47.536 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.536 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2021-03-09T08:53:47.536 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2021-03-09T08:53:47.536 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2021-03-09T08:53:47.536 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2021-03-09T08:53:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2021-03-09T08:53:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2021-03-09T08:53:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2021-03-09T08:53:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T08:53:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T08:53:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2021-03-09T08:53:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.539 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T08:53:47.539 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.539 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2021-03-09T08:53:47.539 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.539 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2021-03-09T08:53:47.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.540 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2021-03-09T08:53:47.540 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2021-03-09T08:53:47.540 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2021-03-09T08:53:47.541 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.41" 2021-03-09T08:53:47.541 INFO:teuthology.orchestra.run.smithi038.stdout: mon election default strategy = 3 2021-03-09T08:53:47.541 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 988e2677-23e9-442a-94d8-290838bc06dd 2021-03-09T08:53:47.541 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.542 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2021-03-09T08:53:47.542 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2021-03-09T08:53:47.542 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.543 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2021-03-09T08:53:47.543 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2021-03-09T08:53:47.543 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.543 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2021-03-09T08:53:47.543 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2021-03-09T08:53:47.544 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.544 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2021-03-09T08:53:47.544 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2021-03-09T08:53:47.544 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2021-03-09T08:53:47.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.545 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2021-03-09T08:53:47.545 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2021-03-09T08:53:47.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.545 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2021-03-09T08:53:47.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.546 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2021-03-09T08:53:47.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.546 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2021-03-09T08:53:47.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.547 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2021-03-09T08:53:47.547 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2021-03-09T08:53:47.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.548 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2021-03-09T08:53:47.548 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2021-03-09T08:53:47.548 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2021-03-09T08:53:47.548 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression algorithm = lz4 2021-03-09T08:53:47.549 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression mode = aggressive 2021-03-09T08:53:47.549 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2021-03-09T08:53:47.549 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 20 2021-03-09T08:53:47.549 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 20 2021-03-09T08:53:47.549 INFO:teuthology.orchestra.run.smithi038.stdout: debug filestore = 20 2021-03-09T08:53:47.550 INFO:teuthology.orchestra.run.smithi038.stdout: debug journal = 20 2021-03-09T08:53:47.550 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgrc = 20 2021-03-09T08:53:47.550 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 20 2021-03-09T08:53:47.550 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 25 2021-03-09T08:53:47.551 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 10 2021-03-09T08:53:47.551 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T08:53:47.552 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2021-03-09T08:53:47.552 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2021-03-09T08:53:47.552 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2021-03-09T08:53:47.553 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2021-03-09T08:53:47.553 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.553 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2021-03-09T08:53:47.553 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2021-03-09T08:53:47.553 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2021-03-09T08:53:47.554 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2021-03-09T08:53:47.554 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2021-03-09T08:53:47.554 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:53:47.554 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:53:47.555 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2021-03-09T08:53:47.556 INFO:teuthology.orchestra.run.smithi038.stdout: debug client = 20 2021-03-09T08:53:47.556 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.557 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2021-03-09T08:53:47.557 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2021-03-09T08:53:47.557 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2021-03-09T08:53:47.558 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2021-03-09T08:53:47.558 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2021-03-09T08:53:47.558 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2021-03-09T08:53:47.559 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2021-03-09T08:53:47.559 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:53:47.559 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2021-03-09T08:53:47.559 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2021-03-09T08:53:47.560 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:53:47.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2021-03-09T08:53:47.560 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2021-03-09T08:53:47.560 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2021-03-09T08:53:47.561 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2021-03-09T08:53:47.561 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2021-03-09T08:53:47.561 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T08:53:47.561 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T08:53:47.562 INFO:teuthology.orchestra.run.smithi038.stdout: debug client = 20 2021-03-09T08:53:47.562 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgrc = 20 2021-03-09T08:53:47.562 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2021-03-09T08:53:47.562 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2021-03-09T08:53:47.562 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.c] 2021-03-09T08:53:47.563 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.b] 2021-03-09T08:53:47.563 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.a] 2021-03-09T08:53:47.564 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.c] 2021-03-09T08:53:47.564 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.b] 2021-03-09T08:53:47.564 INFO:teuthology.orchestra.run.smithi038.stdout:[mds] 2021-03-09T08:53:47.564 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgrc = 20 2021-03-09T08:53:47.567 INFO:teuthology.orchestra.run.smithi041.stdout:[global] 2021-03-09T08:53:47.567 INFO:teuthology.orchestra.run.smithi041.stdout: chdir = "" 2021-03-09T08:53:47.567 INFO:teuthology.orchestra.run.smithi041.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T08:53:47.568 INFO:teuthology.orchestra.run.smithi041.stdout: auth supported = cephx 2021-03-09T08:53:47.568 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.568 INFO:teuthology.orchestra.run.smithi041.stdout: filestore xattr use omap = true 2021-03-09T08:53:47.569 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.569 INFO:teuthology.orchestra.run.smithi041.stdout: mon clock drift allowed = 1.000 2021-03-09T08:53:47.569 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.569 INFO:teuthology.orchestra.run.smithi041.stdout: osd crush chooseleaf type = 0 2021-03-09T08:53:47.569 INFO:teuthology.orchestra.run.smithi041.stdout: auth debug = true 2021-03-09T08:53:47.570 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.570 INFO:teuthology.orchestra.run.smithi041.stdout: ms die on old message = true 2021-03-09T08:53:47.570 INFO:teuthology.orchestra.run.smithi041.stdout: ms die on bug = true 2021-03-09T08:53:47.570 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.571 INFO:teuthology.orchestra.run.smithi041.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T08:53:47.571 INFO:teuthology.orchestra.run.smithi041.stdout: mon pg warn max object skew = 0 2021-03-09T08:53:47.571 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.571 INFO:teuthology.orchestra.run.smithi041.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T08:53:47.572 INFO:teuthology.orchestra.run.smithi041.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T08:53:47.572 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.572 INFO:teuthology.orchestra.run.smithi041.stdout: osd pool default size = 2 2021-03-09T08:53:47.572 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.573 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd allow primary affinity = true 2021-03-09T08:53:47.573 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd allow pg remap = true 2021-03-09T08:53:47.573 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on legacy crush tunables = false 2021-03-09T08:53:47.573 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on crush straw calc version zero = false 2021-03-09T08:53:47.574 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on no sortbitwise = false 2021-03-09T08:53:47.574 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on osd down out interval zero = false 2021-03-09T08:53:47.575 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on too few osds = false 2021-03-09T08:53:47.575 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T08:53:47.575 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T08:53:47.575 INFO:teuthology.orchestra.run.smithi041.stdout: mon_allow_pool_size_one = true 2021-03-09T08:53:47.576 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.576 INFO:teuthology.orchestra.run.smithi041.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T08:53:47.576 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.576 INFO:teuthology.orchestra.run.smithi041.stdout: osd default data pool replay window = 5 2021-03-09T08:53:47.577 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.577 INFO:teuthology.orchestra.run.smithi041.stdout: mon allow pool delete = true 2021-03-09T08:53:47.578 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.578 INFO:teuthology.orchestra.run.smithi041.stdout: mon cluster log file level = debug 2021-03-09T08:53:47.578 INFO:teuthology.orchestra.run.smithi041.stdout: debug asserts on shutdown = true 2021-03-09T08:53:47.578 INFO:teuthology.orchestra.run.smithi041.stdout: mon health detail to clog = false 2021-03-09T08:53:47.579 INFO:teuthology.orchestra.run.smithi041.stdout: mon host = "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.41" 2021-03-09T08:53:47.579 INFO:teuthology.orchestra.run.smithi041.stdout: mon election default strategy = 3 2021-03-09T08:53:47.579 INFO:teuthology.orchestra.run.smithi041.stdout: fsid = 988e2677-23e9-442a-94d8-290838bc06dd 2021-03-09T08:53:47.579 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.580 INFO:teuthology.orchestra.run.smithi041.stdout:[osd] 2021-03-09T08:53:47.580 INFO:teuthology.orchestra.run.smithi041.stdout: osd journal size = 100 2021-03-09T08:53:47.580 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.580 INFO:teuthology.orchestra.run.smithi041.stdout: osd scrub load threshold = 5.0 2021-03-09T08:53:47.581 INFO:teuthology.orchestra.run.smithi041.stdout: osd scrub max interval = 600 2021-03-09T08:53:47.581 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.581 INFO:teuthology.orchestra.run.smithi041.stdout: osd recover clone overlap = true 2021-03-09T08:53:47.581 INFO:teuthology.orchestra.run.smithi041.stdout: osd recovery max chunk = 1048576 2021-03-09T08:53:47.582 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.582 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug shutdown = true 2021-03-09T08:53:47.582 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug op order = true 2021-03-09T08:53:47.582 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug verify stray on activate = true 2021-03-09T08:53:47.583 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.583 INFO:teuthology.orchestra.run.smithi041.stdout: osd open classes on start = true 2021-03-09T08:53:47.583 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug pg log writeout = true 2021-03-09T08:53:47.583 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.584 INFO:teuthology.orchestra.run.smithi041.stdout: osd deep scrub update digest min age = 30 2021-03-09T08:53:47.584 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.584 INFO:teuthology.orchestra.run.smithi041.stdout: osd map max advance = 10 2021-03-09T08:53:47.584 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.585 INFO:teuthology.orchestra.run.smithi041.stdout: journal zero on create = true 2021-03-09T08:53:47.585 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.585 INFO:teuthology.orchestra.run.smithi041.stdout: filestore ondisk finisher threads = 3 2021-03-09T08:53:47.585 INFO:teuthology.orchestra.run.smithi041.stdout: filestore apply finisher threads = 3 2021-03-09T08:53:47.586 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.586 INFO:teuthology.orchestra.run.smithi041.stdout: bdev debug aio = true 2021-03-09T08:53:47.586 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug misdirected ops = true 2021-03-09T08:53:47.586 INFO:teuthology.orchestra.run.smithi041.stdout: bluestore block size = 96636764160 2021-03-09T08:53:47.586 INFO:teuthology.orchestra.run.smithi041.stdout: bluestore compression algorithm = lz4 2021-03-09T08:53:47.587 INFO:teuthology.orchestra.run.smithi041.stdout: bluestore compression mode = aggressive 2021-03-09T08:53:47.587 INFO:teuthology.orchestra.run.smithi041.stdout: bluestore fsck on mount = True 2021-03-09T08:53:47.587 INFO:teuthology.orchestra.run.smithi041.stdout: debug bluefs = 20 2021-03-09T08:53:47.587 INFO:teuthology.orchestra.run.smithi041.stdout: debug bluestore = 20 2021-03-09T08:53:47.588 INFO:teuthology.orchestra.run.smithi041.stdout: debug filestore = 20 2021-03-09T08:53:47.588 INFO:teuthology.orchestra.run.smithi041.stdout: debug journal = 20 2021-03-09T08:53:47.588 INFO:teuthology.orchestra.run.smithi041.stdout: debug mgrc = 20 2021-03-09T08:53:47.588 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 20 2021-03-09T08:53:47.589 INFO:teuthology.orchestra.run.smithi041.stdout: debug osd = 25 2021-03-09T08:53:47.589 INFO:teuthology.orchestra.run.smithi041.stdout: debug rocksdb = 10 2021-03-09T08:53:47.589 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T08:53:47.589 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd full ratio = 0.9 2021-03-09T08:53:47.590 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd nearfull ratio = 0.8 2021-03-09T08:53:47.590 INFO:teuthology.orchestra.run.smithi041.stdout: osd failsafe full ratio = 0.95 2021-03-09T08:53:47.590 INFO:teuthology.orchestra.run.smithi041.stdout: osd objectstore = bluestore 2021-03-09T08:53:47.590 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.591 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr] 2021-03-09T08:53:47.591 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-03-09T08:53:47.591 INFO:teuthology.orchestra.run.smithi041.stdout: debug mgr = 20 2021-03-09T08:53:47.591 INFO:teuthology.orchestra.run.smithi041.stdout: debug mon = 20 2021-03-09T08:53:47.591 INFO:teuthology.orchestra.run.smithi041.stdout: debug auth = 20 2021-03-09T08:53:47.592 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:53:47.592 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:53:47.592 INFO:teuthology.orchestra.run.smithi041.stdout: mgr/telemetry/nag = false 2021-03-09T08:53:47.592 INFO:teuthology.orchestra.run.smithi041.stdout: debug client = 20 2021-03-09T08:53:47.593 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.593 INFO:teuthology.orchestra.run.smithi041.stdout:[mon] 2021-03-09T08:53:47.593 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-03-09T08:53:47.593 INFO:teuthology.orchestra.run.smithi041.stdout: debug mon = 20 2021-03-09T08:53:47.594 INFO:teuthology.orchestra.run.smithi041.stdout: debug paxos = 20 2021-03-09T08:53:47.594 INFO:teuthology.orchestra.run.smithi041.stdout: debug auth = 20 2021-03-09T08:53:47.594 INFO:teuthology.orchestra.run.smithi041.stdout: mon data avail warn = 5 2021-03-09T08:53:47.594 INFO:teuthology.orchestra.run.smithi041.stdout: mon mgr mkfs grace = 240 2021-03-09T08:53:47.595 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:53:47.595 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd reporter subtree level = osd 2021-03-09T08:53:47.595 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd prime pg temp = true 2021-03-09T08:53:47.596 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:53:47.596 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-03-09T08:53:47.596 INFO:teuthology.orchestra.run.smithi041.stdout:[client] 2021-03-09T08:53:47.596 INFO:teuthology.orchestra.run.smithi041.stdout: rgw cache enabled = true 2021-03-09T08:53:47.597 INFO:teuthology.orchestra.run.smithi041.stdout: rgw enable ops log = true 2021-03-09T08:53:47.597 INFO:teuthology.orchestra.run.smithi041.stdout: rgw enable usage log = true 2021-03-09T08:53:47.597 INFO:teuthology.orchestra.run.smithi041.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T08:53:47.597 INFO:teuthology.orchestra.run.smithi041.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T08:53:47.598 INFO:teuthology.orchestra.run.smithi041.stdout: debug client = 20 2021-03-09T08:53:47.598 INFO:teuthology.orchestra.run.smithi041.stdout: debug mgrc = 20 2021-03-09T08:53:47.598 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-03-09T08:53:47.598 INFO:teuthology.orchestra.run.smithi041.stdout:[mon.a] 2021-03-09T08:53:47.599 INFO:teuthology.orchestra.run.smithi041.stdout:[mon.c] 2021-03-09T08:53:47.599 INFO:teuthology.orchestra.run.smithi041.stdout:[mon.b] 2021-03-09T08:53:47.599 INFO:teuthology.orchestra.run.smithi041.stdout:[mds.a] 2021-03-09T08:53:47.599 INFO:teuthology.orchestra.run.smithi041.stdout:[mds.c] 2021-03-09T08:53:47.600 INFO:teuthology.orchestra.run.smithi041.stdout:[mds.b] 2021-03-09T08:53:47.600 INFO:teuthology.orchestra.run.smithi041.stdout:[mds] 2021-03-09T08:53:47.600 INFO:teuthology.orchestra.run.smithi041.stdout: debug mgrc = 20 2021-03-09T08:53:47.601 INFO:tasks.ceph:Creating admin key on mon.a... 2021-03-09T08:53:47.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-03-09T08:53:47.713 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-09T08:53:47.714 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:47.714 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-03-09T08:53:47.721 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:47.722 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-03-09T08:53:47.773 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2021-03-09T08:53:47.773 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:47.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T08:53:47.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T08:53:47.840 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:53:47.841 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:47.893 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi041.front.sepia.ceph.com 2021-03-09T08:53:47.893 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:47.894 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T08:53:47.894 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T08:53:47.921 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:53:47.921 DEBUG:teuthology.orchestra.run.smithi041:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:47.967 INFO:tasks.ceph:Setting up mon nodes... 2021-03-09T08:53:47.968 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-09T08:53:47.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T08:53:48.072 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T08:53:48.078 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-03-09T08:53:48.182 INFO:teuthology.orchestra.run.smithi041.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-03-09T08:53:48.189 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/mgr/ceph-z && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.z /var/lib/ceph/mgr/ceph-z/keyring 2021-03-09T08:53:48.289 INFO:teuthology.orchestra.run.smithi041.stdout:creating /var/lib/ceph/mgr/ceph-z/keyring 2021-03-09T08:53:48.296 INFO:tasks.ceph:Setting up mds nodes... 2021-03-09T08:53:48.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2021-03-09T08:53:48.399 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2021-03-09T08:53:48.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2021-03-09T08:53:48.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2021-03-09T08:53:48.571 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2021-03-09T08:53:48.577 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2021-03-09T08:53:48.600 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2021-03-09T08:53:48.705 INFO:teuthology.orchestra.run.smithi041.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2021-03-09T08:53:48.713 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2021-03-09T08:53:48.728 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-09T08:53:48.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-03-09T08:53:48.824 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-03-09T08:53:48.842 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-03-09T08:53:48.938 INFO:teuthology.orchestra.run.smithi041.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-03-09T08:53:48.954 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-09T08:53:48.955 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi041.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2021-03-09T08:53:48.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-03-09T08:53:48.973 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2021-03-09T08:53:48.974 INFO:tasks.ceph:role: osd.0 2021-03-09T08:53:48.974 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2021-03-09T08:53:48.975 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T08:53:50.371 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:53:50.372 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:53:50.372 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T08:53:50.372 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:53:50.373 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:53:50.373 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T08:53:50.373 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:53:50.373 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:53:50.373 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:53:50.375 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2021-03-09T08:53:50.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-03-09T08:53:50.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-03-09T08:53:50.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T08:53:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T08:53:50.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:50.613 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:53:50.610+0000 7f37f8a8df00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-03-09T08:53:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:53:50.610+0000 7f37f8a8df00 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-03-09T08:53:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:53:50.610+0000 7f37f8a8df00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T08:53:50.615 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:53:50.610+0000 7f37f8a8df00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T08:53:50.615 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:53:50.610+0000 7f37f8a8df00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T08:53:50.617 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:53:50.614+0000 7f37f8a8df00 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-03-09T08:53:53.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-03-09T08:53:53.617 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-03-09T08:53:53.633 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T08:53:53.634 INFO:tasks.ceph:role: osd.1 2021-03-09T08:53:53.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi041.front.sepia.ceph.com 2021-03-09T08:53:53.634 DEBUG:teuthology.orchestra.run.smithi041:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T08:53:54.155 INFO:teuthology.orchestra.run.smithi041.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:53:54.156 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:53:54.156 INFO:teuthology.orchestra.run.smithi041.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T08:53:54.156 INFO:teuthology.orchestra.run.smithi041.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:53:54.156 INFO:teuthology.orchestra.run.smithi041.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:53:54.157 INFO:teuthology.orchestra.run.smithi041.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T08:53:54.157 INFO:teuthology.orchestra.run.smithi041.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:53:54.157 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:53:54.157 INFO:teuthology.orchestra.run.smithi041.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:53:54.159 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi041.front.sepia.ceph.com -o noatime 2021-03-09T08:53:54.159 DEBUG:teuthology.orchestra.run.smithi041:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2021-03-09T08:53:54.234 DEBUG:teuthology.orchestra.run.smithi041:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-03-09T08:53:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T08:53:54.247 INFO:teuthology.orchestra.run.smithi041.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T08:53:54.248 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-03-09T08:53:54.304 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T08:53:54.305 INFO:tasks.ceph:role: osd.2 2021-03-09T08:53:54.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi041.front.sepia.ceph.com 2021-03-09T08:53:54.305 DEBUG:teuthology.orchestra.run.smithi041:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-09T08:53:54.817 INFO:teuthology.orchestra.run.smithi041.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:53:54.817 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:53:54.818 INFO:teuthology.orchestra.run.smithi041.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T08:53:54.818 INFO:teuthology.orchestra.run.smithi041.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:53:54.818 INFO:teuthology.orchestra.run.smithi041.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:53:54.818 INFO:teuthology.orchestra.run.smithi041.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T08:53:54.819 INFO:teuthology.orchestra.run.smithi041.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:53:54.819 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:53:54.819 INFO:teuthology.orchestra.run.smithi041.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:53:54.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi041.front.sepia.ceph.com -o noatime 2021-03-09T08:53:54.822 DEBUG:teuthology.orchestra.run.smithi041:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2021-03-09T08:53:54.888 DEBUG:teuthology.orchestra.run.smithi041:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-03-09T08:53:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T08:53:54.903 INFO:teuthology.orchestra.run.smithi041.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T08:53:54.903 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-03-09T08:53:54.958 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2021-03-09T08:53:54.958 INFO:tasks.ceph:role: osd.3 2021-03-09T08:53:54.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi041.front.sepia.ceph.com 2021-03-09T08:53:54.959 DEBUG:teuthology.orchestra.run.smithi041:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-09T08:53:55.463 INFO:teuthology.orchestra.run.smithi041.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:53:55.463 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:53:55.464 INFO:teuthology.orchestra.run.smithi041.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T08:53:55.464 INFO:teuthology.orchestra.run.smithi041.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:53:55.464 INFO:teuthology.orchestra.run.smithi041.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:53:55.464 INFO:teuthology.orchestra.run.smithi041.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T08:53:55.465 INFO:teuthology.orchestra.run.smithi041.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:53:55.465 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:53:55.465 INFO:teuthology.orchestra.run.smithi041.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:53:55.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi041.front.sepia.ceph.com -o noatime 2021-03-09T08:53:55.467 DEBUG:teuthology.orchestra.run.smithi041:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2021-03-09T08:53:55.518 DEBUG:teuthology.orchestra.run.smithi041:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-03-09T08:53:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T08:53:55.531 INFO:teuthology.orchestra.run.smithi041.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T08:53:55.532 DEBUG:teuthology.orchestra.run.smithi041:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:55.686 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:55.687+0000 7fc860de4f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-03-09T08:53:55.688 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:55.687+0000 7fc860de4f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-03-09T08:53:55.688 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:55.687+0000 7fc860de4f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T08:53:55.688 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:55.687+0000 7fc860de4f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T08:53:55.689 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:55.687+0000 7fc860de4f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T08:53:55.692 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:55.691+0000 7fc860de4f00 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-03-09T08:53:58.698 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-03-09T08:53:58.712 DEBUG:teuthology.orchestra.run.smithi041:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:53:58.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:58.847+0000 7f8a36c22f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-03-09T08:53:58.850 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:58.851+0000 7f8a36c22f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-03-09T08:53:58.850 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:58.851+0000 7f8a36c22f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T08:53:58.850 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:58.851+0000 7f8a36c22f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T08:53:58.851 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:58.851+0000 7f8a36c22f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T08:53:58.851 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:53:58.851+0000 7f8a36c22f00 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-03-09T08:54:01.852 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-09T08:54:01.864 DEBUG:teuthology.orchestra.run.smithi041:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:54:02.001 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:01.999+0000 7fc0ea958f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-03-09T08:54:02.002 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:01.999+0000 7fc0ea958f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-03-09T08:54:02.002 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:01.999+0000 7fc0ea958f00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T08:54:02.002 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:01.999+0000 7fc0ea958f00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T08:54:02.003 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:01.999+0000 7fc0ea958f00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T08:54:02.003 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:02.003+0000 7fc0ea958f00 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-03-09T08:54:04.984 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-03-09T08:54:04.997 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-09T08:54:04.998 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:54:04.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-03-09T08:54:05.015 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:54:05.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2021-03-09T08:54:05.073 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:54:05.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2021-03-09T08:54:05.129 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:54:05.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-03-09T08:54:05.185 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.185 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2021-03-09T08:54:05.201 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.201 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/mgr/ceph-z/keyring of=/dev/stdout 2021-03-09T08:54:05.259 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.260 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2021-03-09T08:54:05.315 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.316 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-03-09T08:54:05.372 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.372 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-03-09T08:54:05.426 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.426 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-03-09T08:54:05.483 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:54:05.484 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2021-03-09T08:54:05.492 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:05.492 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2021-03-09T08:54:05.531 INFO:tasks.ceph:Adding keys to all mons... 2021-03-09T08:54:05.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T08:54:05.536 DEBUG:teuthology.orchestra.run.smithi041:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T08:54:05.575 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2021-03-09T08:54:05.576 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dgjhlUBBAAlqvK7Fy2QwfEDKRClgEVYw== 2021-03-09T08:54:05.576 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.a] 2021-03-09T08:54:05.576 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dgUVrUFxAAPRjQLZBPkUH4Y4NUatsHGA== 2021-03-09T08:54:05.577 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.c] 2021-03-09T08:54:05.577 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dgmmMSIhAAIWeWlWqIh6CWwiPDD/6A2w== 2021-03-09T08:54:05.577 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2021-03-09T08:54:05.577 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCeN0dgxyWYJBAA95kv0l2I6CfSb0PUmn8lTg== 2021-03-09T08:54:05.578 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.y] 2021-03-09T08:54:05.578 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dg/WwCCxAAJIwKQlk/Ps29bBS7hDsm/w== 2021-03-09T08:54:05.578 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.z] 2021-03-09T08:54:05.579 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dgtoxiERAAPB321HHpPWhFBXY5wKWKWg== 2021-03-09T08:54:05.579 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.b] 2021-03-09T08:54:05.579 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dg0pUyKhAABZjLkkmr8YXaN3ZgaOqsAg== 2021-03-09T08:54:05.579 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2021-03-09T08:54:05.580 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCjN0dg1JQJKRAAcIiZ34aGrTnLcaYuSTFGXw== 2021-03-09T08:54:05.580 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2021-03-09T08:54:05.580 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCmN0dg+XvDMhAAdywymN3z85lDPiDsHEJeqA== 2021-03-09T08:54:05.580 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.3] 2021-03-09T08:54:05.581 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCqN0dgkgkpABAA42c2m1SlDv+bSuRLbHJFqQ== 2021-03-09T08:54:05.581 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2021-03-09T08:54:05.581 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dgQuswMRAAZDdc89we2VHGFN5OZe501w== 2021-03-09T08:54:05.581 INFO:teuthology.orchestra.run.smithi038.stdout:[client.1] 2021-03-09T08:54:05.582 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCcN0dgH7oNOBAAKpdIzv+gTko+Qq0uz6rq+w== 2021-03-09T08:54:05.586 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr.x] 2021-03-09T08:54:05.586 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dgjhlUBBAAlqvK7Fy2QwfEDKRClgEVYw== 2021-03-09T08:54:05.587 INFO:teuthology.orchestra.run.smithi041.stdout:[mds.a] 2021-03-09T08:54:05.587 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dgUVrUFxAAPRjQLZBPkUH4Y4NUatsHGA== 2021-03-09T08:54:05.587 INFO:teuthology.orchestra.run.smithi041.stdout:[mds.c] 2021-03-09T08:54:05.587 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dgmmMSIhAAIWeWlWqIh6CWwiPDD/6A2w== 2021-03-09T08:54:05.588 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.0] 2021-03-09T08:54:05.588 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCeN0dgxyWYJBAA95kv0l2I6CfSb0PUmn8lTg== 2021-03-09T08:54:05.588 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr.y] 2021-03-09T08:54:05.588 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dg/WwCCxAAJIwKQlk/Ps29bBS7hDsm/w== 2021-03-09T08:54:05.589 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr.z] 2021-03-09T08:54:05.589 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dgtoxiERAAPB321HHpPWhFBXY5wKWKWg== 2021-03-09T08:54:05.589 INFO:teuthology.orchestra.run.smithi041.stdout:[mds.b] 2021-03-09T08:54:05.589 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dg0pUyKhAABZjLkkmr8YXaN3ZgaOqsAg== 2021-03-09T08:54:05.590 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.1] 2021-03-09T08:54:05.590 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCjN0dg1JQJKRAAcIiZ34aGrTnLcaYuSTFGXw== 2021-03-09T08:54:05.590 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.2] 2021-03-09T08:54:05.591 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCmN0dg+XvDMhAAdywymN3z85lDPiDsHEJeqA== 2021-03-09T08:54:05.591 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.3] 2021-03-09T08:54:05.591 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCqN0dgkgkpABAA42c2m1SlDv+bSuRLbHJFqQ== 2021-03-09T08:54:05.591 INFO:teuthology.orchestra.run.smithi041.stdout:[client.0] 2021-03-09T08:54:05.591 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dgQuswMRAAZDdc89we2VHGFN5OZe501w== 2021-03-09T08:54:05.592 INFO:teuthology.orchestra.run.smithi041.stdout:[client.1] 2021-03-09T08:54:05.592 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQCcN0dgH7oNOBAAKpdIzv+gTko+Qq0uz6rq+w== 2021-03-09T08:54:05.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:54:05.629 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:54:05.744 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T08:54:05.750 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T08:54:05.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T08:54:05.872 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T08:54:05.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:05.986 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:54:06.099 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:54:06.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:54:06.215 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:54:06.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T08:54:06.333 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-03-09T08:54:06.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.449 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.552 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.657 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:54:06.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:54:06.765 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:54:06.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:54:06.883 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:54:06.992 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-09T08:54:06.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-03-09T08:54:07.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T08:54:07.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-03-09T08:54:07.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-03-09T08:54:07.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T08:54:07.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-03-09T08:54:07.465 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-03-09T08:54:07.484 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T08:54:07.673 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-03-09T08:54:07.687 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:54:07.690 DEBUG:teuthology.orchestra.run.smithi041:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:54:07.735 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-03-09T08:54:07.735 INFO:tasks.ceph.mon.a:Restarting daemon 2021-03-09T08:54:07.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-03-09T08:54:07.740 INFO:tasks.ceph.mon.a:Started 2021-03-09T08:54:07.741 INFO:tasks.ceph.mon.c:Restarting daemon 2021-03-09T08:54:07.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-03-09T08:54:07.743 INFO:tasks.ceph.mon.c:Started 2021-03-09T08:54:07.744 INFO:tasks.ceph.mon.b:Restarting daemon 2021-03-09T08:54:07.745 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-03-09T08:54:07.778 INFO:tasks.ceph.mon.b:Started 2021-03-09T08:54:07.779 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-03-09T08:54:07.780 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-03-09T08:54:07.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-03-09T08:54:07.783 INFO:tasks.ceph.mgr.x:Started 2021-03-09T08:54:07.784 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-03-09T08:54:07.784 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-03-09T08:54:07.787 INFO:tasks.ceph.mgr.y:Started 2021-03-09T08:54:07.787 INFO:tasks.ceph.mgr.z:Restarting daemon 2021-03-09T08:54:07.788 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i z 2021-03-09T08:54:07.790 INFO:tasks.ceph.mgr.z:Started 2021-03-09T08:54:07.791 INFO:tasks.ceph:Setting crush tunables to default 2021-03-09T08:54:07.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd crush tunables default 2021-03-09T08:54:08.054 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.050+0000 7fc71ffff700 1 Processor -- start 2021-03-09T08:54:08.054 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.050+0000 7fc71ffff700 1 -- start start 2021-03-09T08:54:08.057 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71ffff700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:08.057 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71ffff700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:08.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71ffff700 1 -- --> v1:172.21.15.41:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7201391d0 con 0x7fc72005c8f0 2021-03-09T08:54:08.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71ffff700 1 -- --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72009da10 con 0x7fc720065c80 2021-03-09T08:54:08.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71ffff700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc720067780 con 0x7fc720068410 2021-03-09T08:54:08.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:08.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71e7fc700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:08.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46402/0 (socket says 172.21.15.38:46402) 2021-03-09T08:54:08.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 learned_addr learned my addr 172.21.15.38:0/1095470350 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:08.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:54:08.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:08.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:08.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:08.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:54:08.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:08.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:08.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:08.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:08.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:08.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:54:08.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:08.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:08.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:54:08.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:08.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:08.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:54:08.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:54:08.664 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:08.664 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:08.664 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:54:08.665 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:08.665 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:08.665 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:54:08.665 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:08.666 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:08.666 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:54:08.666 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:08.662+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:54:09.468 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:09.468 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:09.468 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:54:09.469 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:09.469 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:09.469 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:54:09.469 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:54:09.470 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:09.470 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:09.470 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:09.466+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:54:11.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:11.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc7201396a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:11.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 msgr2=0x7fc720139b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:11.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc720065c80 0x7fc720139b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:11.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 >> v1:172.21.15.41:6789/0 conn(0x7fc72005c8f0 legacy=0x7fc720067070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:11.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:11.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 --> v1:172.21.15.38:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc720139400 con 0x7fc7040033a0 2021-03-09T08:54:11.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72009da10 con 0x7fc704007030 2021-03-09T08:54:11.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:11.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:54:11.063 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:11.063 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:11.064 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.258+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:54:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.258+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:11.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:54:11.663 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.662+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:11.664 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.662+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:54:11.664 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.662+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:11.664 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.662+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:11.665 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:11.662+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:54:12.467 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:12.466+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:12.468 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:12.466+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:54:12.468 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:12.466+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:12.468 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:12.466+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:12.469 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:12.466+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:54:14.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:14.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc704007030 0x7fc7040094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.058 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 >> v1:172.21.15.38:6789/0 conn(0x7fc7040033a0 legacy=0x7fc704005820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 reap_dead start 2021-03-09T08:54:14.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.059 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 0x7fc704005820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72009da10 con 0x7fc70400a5e0 2021-03-09T08:54:14.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc720067780 con 0x7fc70400a140 2021-03-09T08:54:14.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71cff9700 1 -- 172.21.15.38:0/1095470350 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc704005df0 con 0x7fc720068410 2021-03-09T08:54:14.060 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71effd700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 0x7fc704005820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.061 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71effd700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc704009c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:54:14.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 msgr2=0x7fc704005820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:54:14.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71effd700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc704009c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:14.062 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 msgr2=0x7fc704005820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:14.063 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 0x7fc704005820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:14.063 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71effd700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:14.063 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 0x7fc704005820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:14.063 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71effd700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:54:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71e7fc700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:54:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.054+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71effd700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.38:3300/0 conn(0x7fc70400a5e0 0x7fc704005820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 msgr2=0x7fc704009c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 msgr2=0x7fc7040094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:14.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 -- 172.21.15.38:0/1095470350 --> v2:172.21.15.38:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70400ac00 con 0x7fc70400a5e0 2021-03-09T08:54:14.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71effd700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:54:14.262 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc70400a5e0 0x7fc704005820 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc7200a9ae0 tx=0x7fc718008040).ready entity=mon.0 client_cookie=b1ec84ca7654c5bb server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71dffb700 1 -- 172.21.15.38:0/1095470350 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718016b00 con 0x7fc70400a5e0 2021-03-09T08:54:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71dffb700 1 -- 172.21.15.38:0/1095470350 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71800c040 con 0x7fc70400a5e0 2021-03-09T08:54:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71dffb700 1 -- 172.21.15.38:0/1095470350 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718016e70 con 0x7fc70400a5e0 2021-03-09T08:54:14.264 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 -- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc70400a5e0 msgr2=0x7fc704005820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.264 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc70400a5e0 0x7fc704005820 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc7200a9ae0 tx=0x7fc718008040).stop 2021-03-09T08:54:14.265 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 -- 172.21.15.38:0/1095470350 shutdown_connections 2021-03-09T08:54:14.265 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc70400a5e0 0x7fc704005820 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.265 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 --2- 172.21.15.38:0/1095470350 >> v2:172.21.15.41:3300/0 conn(0x7fc70400a140 0x7fc7040094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.265 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 --2- 172.21.15.38:0/1095470350 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc720068410 0x7fc704009c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 -- 172.21.15.38:0/1095470350 >> 172.21.15.38:0/1095470350 conn(0x7fc7200649c0 msgr2=0x7fc720068e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 -- 172.21.15.38:0/1095470350 shutdown_connections 2021-03-09T08:54:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.258+0000 7fc71ffff700 1 -- 172.21.15.38:0/1095470350 wait complete. 2021-03-09T08:54:14.269 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.262+0000 7fc71ffff700 1 Processor -- start 2021-03-09T08:54:14.269 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.262+0000 7fc71ffff700 1 -- start start 2021-03-09T08:54:14.269 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7fc70400a140 0x7fc7200a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.269 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc70400a5e0 0x7fc7200a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.270 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 0x7fc7200a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.270 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200a0680 con 0x7fc720068410 2021-03-09T08:54:14.270 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200a0920 con 0x7fc70400a140 2021-03-09T08:54:14.270 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7201327d0 con 0x7fc70400a5e0 2021-03-09T08:54:14.271 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 0x7fc7200a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.271 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 0x7fc7200a4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46490/0 (socket says 172.21.15.38:46490) 2021-03-09T08:54:14.271 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71f7fe700 1 -- 172.21.15.38:0/2552029437 learned_addr learned my addr 172.21.15.38:0/2552029437 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:14.271 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc70400a5e0 0x7fc7200a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.272 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71effd700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7fc70400a140 0x7fc7200a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.272 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 -- 172.21.15.38:0/2552029437 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7fc70400a140 msgr2=0x7fc7200a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.272 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7fc70400a140 0x7fc7200a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.272 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 -- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 msgr2=0x7fc7200a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.272 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 0x7fc7200a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.273 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 -- 172.21.15.38:0/2552029437 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200a2af0 con 0x7fc70400a5e0 2021-03-09T08:54:14.273 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71f7fe700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 0x7fc7200a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:54:14.273 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71e7fc700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc70400a5e0 0x7fc7200a3270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc708006bf0 tx=0x7fc70801c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:14.273 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70801f030 con 0x7fc70400a5e0 2021-03-09T08:54:14.274 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc708020040 con 0x7fc70400a5e0 2021-03-09T08:54:14.274 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7200a2ce0 con 0x7fc70400a5e0 2021-03-09T08:54:14.274 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708016ae0 con 0x7fc70400a5e0 2021-03-09T08:54:14.274 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7200a4c70 con 0x7fc70400a5e0 2021-03-09T08:54:14.275 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fc708012040 con 0x7fc70400a5e0 2021-03-09T08:54:14.275 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T08:54:14.275 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T08:54:14.276 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T08:54:14.276 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 4 mgrc reconnect No active mgr available yet 2021-03-09T08:54:14.276 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.266+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc708018e40 con 0x7fc70400a5e0 2021-03-09T08:54:14.276 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.270+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72009cca0 con 0x7fc70400a5e0 2021-03-09T08:54:14.277 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.274+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fc708018450 con 0x7fc70400a5e0 2021-03-09T08:54:14.354 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.350+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc7200a7ff0 con 0x7fc70400a5e0 2021-03-09T08:54:14.379 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc716ffd700 1 -- 172.21.15.38:0/2552029437 <== mon.2 v2:172.21.15.38:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc7080185f0 con 0x7fc70400a5e0 2021-03-09T08:54:14.380 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 10 mgrc shutdown 2021-03-09T08:54:14.381 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc70400a5e0 msgr2=0x7fc7200a3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.381 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc70400a5e0 0x7fc7200a3270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc708006bf0 tx=0x7fc70801c040).stop 2021-03-09T08:54:14.382 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 shutdown_connections 2021-03-09T08:54:14.382 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc720068410 0x7fc7200a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.382 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7fc70400a140 0x7fc7200a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.383 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 --2- 172.21.15.38:0/2552029437 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7fc70400a5e0 0x7fc7200a3270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.383 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 >> 172.21.15.38:0/2552029437 conn(0x7fc7200649c0 msgr2=0x7fc7201396a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:14.383 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 shutdown_connections 2021-03-09T08:54:14.384 INFO:teuthology.orchestra.run.smithi038.stderr:2021-03-09T08:54:14.378+0000 7fc71ffff700 1 -- 172.21.15.38:0/2552029437 wait complete. 2021-03-09T08:54:14.384 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2021-03-09T08:54:14.401 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-03-09T08:54:14.403 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2021-03-09T08:54:14.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-03-09T08:54:14.417 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:14.418 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-03-09T08:54:14.430 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:14.430 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-03-09T08:54:14.486 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-03-09T08:54:14.487 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-03-09T08:54:14.543 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster ceph osd new dcfa5872-e85a-4aa3-a0c7-153e32b0fb29 0 2021-03-09T08:54:14.775 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 Processor -- start 2021-03-09T08:54:14.775 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 -- start start 2021-03-09T08:54:14.777 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f8ed0061120 0x7f8ed0066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.778 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f8ed0067d30 0x7f8ed013cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.778 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 -- --> v1:172.21.15.41:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed013c730 con 0x7f8ed00aa090 2021-03-09T08:54:14.778 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 -- --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed0061500 con 0x7f8ed0067d30 2021-03-09T08:54:14.778 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 -- --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed0066f50 con 0x7f8ed0061120 2021-03-09T08:54:14.779 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f8ed0061120 0x7f8ed0066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.779 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f8ed0061120 0x7f8ed0066700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.41:3300/0 says I am v2:172.21.15.41:42228/0 (socket says 172.21.15.41:42228) 2021-03-09T08:54:14.779 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 -- 172.21.15.41:0/2638078338 learned_addr learned my addr 172.21.15.41:0/2638078338 (peer_addr_for_me v2:172.21.15.41:0/0) 2021-03-09T08:54:14.779 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed49ef700 1 --1- 172.21.15.41:0/2638078338 >> v1:172.21.15.41:6789/0 conn(0x7f8ed00aa090 0x7f8ed00631c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.41:6789/0 says I am v1:172.21.15.41:37436/0 (socket says 172.21.15.41:37436) 2021-03-09T08:54:14.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ec7fff700 1 --2- 172.21.15.41:0/2638078338 >> v2:172.21.15.38:3300/0 conn(0x7f8ed0067d30 0x7f8ed013cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 -- 172.21.15.41:0/2638078338 >> v2:172.21.15.38:3300/0 conn(0x7f8ed0067d30 msgr2=0x7f8ed013cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:14.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 --2- 172.21.15.41:0/2638078338 >> v2:172.21.15.38:3300/0 conn(0x7f8ed0067d30 0x7f8ed013cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 -- 172.21.15.41:0/2638078338 >> v1:172.21.15.41:6789/0 conn(0x7f8ed00aa090 legacy=0x7f8ed00631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 -- 172.21.15.41:0/2638078338 --> v2:172.21.15.41:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0139050 con 0x7f8ed0061120 2021-03-09T08:54:14.781 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ec7fff700 1 --2- 172.21.15.41:0/2638078338 >> v2:172.21.15.38:3300/0 conn(0x7f8ed0067d30 0x7f8ed013cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-03-09T08:54:14.781 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecffff700 1 --2- 172.21.15.41:0/2638078338 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0066700 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f8ed0065ad0 tx=0x7f8ec0006ea0).ready entity=mon.1 client_cookie=98b28bd5812d1463 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:14.781 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecf7fe700 1 -- 172.21.15.41:0/2638078338 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00098d0 con 0x7f8ed0061120 2021-03-09T08:54:14.781 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecf7fe700 1 -- 172.21.15.41:0/2638078338 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0009a30 con 0x7f8ed0061120 2021-03-09T08:54:14.782 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ecf7fe700 1 -- 172.21.15.41:0/2638078338 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0009db0 con 0x7f8ed0061120 2021-03-09T08:54:14.782 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 -- 172.21.15.41:0/2638078338 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 msgr2=0x7f8ed0066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.782 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.775+0000 7f8ed6452700 1 --2- 172.21.15.41:0/2638078338 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0066700 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f8ed0065ad0 tx=0x7f8ec0006ea0).stop 2021-03-09T08:54:14.782 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- 172.21.15.41:0/2638078338 shutdown_connections 2021-03-09T08:54:14.782 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 --2- 172.21.15.41:0/2638078338 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0066700 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.783 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 --2- 172.21.15.41:0/2638078338 >> v2:172.21.15.38:3300/0 conn(0x7f8ed0067d30 0x7f8ed013cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.783 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- 172.21.15.41:0/2638078338 >> 172.21.15.41:0/2638078338 conn(0x7f8ed00628b0 msgr2=0x7f8ed0064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:14.783 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- 172.21.15.41:0/2638078338 shutdown_connections 2021-03-09T08:54:14.783 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- 172.21.15.41:0/2638078338 wait complete. 2021-03-09T08:54:14.786 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 Processor -- start 2021-03-09T08:54:14.786 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- start start 2021-03-09T08:54:14.787 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0137d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.787 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ed00aa090 0x7f8ed0132680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.787 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ed0136200 0x7f8ed0136680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:14.787 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed014bd70 con 0x7f8ed0136200 2021-03-09T08:54:14.788 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0132b70 con 0x7f8ed0061120 2021-03-09T08:54:14.788 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0134490 con 0x7f8ed00aa090 2021-03-09T08:54:14.788 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0137d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.788 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0137d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.41:3300/0 says I am v2:172.21.15.41:42234/0 (socket says 172.21.15.41:42234) 2021-03-09T08:54:14.788 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 -- 172.21.15.41:0/8964775 learned_addr learned my addr 172.21.15.41:0/8964775 (peer_addr_for_me v2:172.21.15.41:0/0) 2021-03-09T08:54:14.789 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ec7fff700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ed00aa090 0x7f8ed0132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.789 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed49ef700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ed0136200 0x7f8ed0136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:14.789 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 -- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ed00aa090 msgr2=0x7f8ed0132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.789 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ed00aa090 0x7f8ed0132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.790 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 -- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ed0136200 msgr2=0x7f8ed0136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.790 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ed0136200 0x7f8ed0136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.790 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 -- 172.21.15.41:0/8964775 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0136b70 con 0x7f8ed0061120 2021-03-09T08:54:14.790 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed49ef700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ed0136200 0x7f8ed0136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-03-09T08:54:14.790 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ecffff700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0137d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8ec0011920 tx=0x7f8ec0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:14.791 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0011e80 con 0x7f8ed0061120 2021-03-09T08:54:14.791 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec001d430 con 0x7f8ed0061120 2021-03-09T08:54:14.791 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec001d7b0 con 0x7f8ed0061120 2021-03-09T08:54:14.791 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed0138850 con 0x7f8ed0061120 2021-03-09T08:54:14.792 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.779+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed014c1f0 con 0x7f8ed0061120 2021-03-09T08:54:14.792 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f8ec0024030 con 0x7f8ed0061120 2021-03-09T08:54:14.792 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8eccff9700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T08:54:14.792 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8eccff9700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T08:54:14.792 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8eccff9700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T08:54:14.793 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8eccff9700 4 mgrc reconnect No active mgr available yet 2021-03-09T08:54:14.793 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8ec0027b70 con 0x7f8ed0061120 2021-03-09T08:54:14.793 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.783+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea8004d60 con 0x7f8ed0061120 2021-03-09T08:54:14.793 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.787+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f8ec002c060 con 0x7f8ed0061120 2021-03-09T08:54:14.869 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.867+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dcfa5872-e85a-4aa3-a0c7-153e32b0fb29", "id": 0} v 0) v1 -- 0x7f8ea80052e0 con 0x7f8ed0061120 2021-03-09T08:54:14.890 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8eccff9700 1 -- 172.21.15.41:0/8964775 <== mon.1 v2:172.21.15.41:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dcfa5872-e85a-4aa3-a0c7-153e32b0fb29", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8ec0022070 con 0x7f8ed0061120 2021-03-09T08:54:14.891 INFO:teuthology.orchestra.run.smithi041.stdout:0 2021-03-09T08:54:14.892 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 10 mgrc shutdown 2021-03-09T08:54:14.893 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 msgr2=0x7f8ed0137d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:14.893 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0137d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8ec0011920 tx=0x7f8ec0016000).stop 2021-03-09T08:54:14.893 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 shutdown_connections 2021-03-09T08:54:14.893 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ed0136200 0x7f8ed0136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.894 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ed0061120 0x7f8ed0137d50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.894 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 --2- 172.21.15.41:0/8964775 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ed00aa090 0x7f8ed0132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:14.894 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 >> 172.21.15.41:0/8964775 conn(0x7f8ed00628b0 msgr2=0x7f8ed00a18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:14.894 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 shutdown_connections 2021-03-09T08:54:14.894 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:14.891+0000 7f8ed6452700 1 -- 172.21.15.41:0/8964775 wait complete. 2021-03-09T08:54:14.911 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster ceph osd new a847843b-12ff-4420-a4ea-c4c413008de2 1 2021-03-09T08:54:15.071 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 Processor -- start 2021-03-09T08:54:15.072 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 -- start start 2021-03-09T08:54:15.075 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f1ab8099190 0x7f1ab809bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.075 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f1ab81398c0 0x7f1ab8139d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 -- --> v1:172.21.15.41:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab80a7e30 con 0x7f1ab805ca90 2021-03-09T08:54:15.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 -- --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab805ce70 con 0x7f1ab8099190 2021-03-09T08:54:15.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abfb9e700 1 -- --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab809a7c0 con 0x7f1ab81398c0 2021-03-09T08:54:15.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abd93a700 1 --1- >> v1:172.21.15.41:6789/0 conn(0x7f1ab805ca90 0x7f1ab80a7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.41:6789/0 says I am v1:172.21.15.41:37444/0 (socket says 172.21.15.41:37444) 2021-03-09T08:54:15.077 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abe13b700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f1ab81398c0 0x7f1ab8139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.077 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abd93a700 1 -- 172.21.15.41:0/4150881304 learned_addr learned my addr 172.21.15.41:0/4150881304 (peer_addr_for_me v1:172.21.15.41:0/0) 2021-03-09T08:54:15.077 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abd139700 1 --2- 172.21.15.41:0/4150881304 >> v2:172.21.15.38:3300/0 conn(0x7f1ab8099190 0x7f1ab809bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.077 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abc938700 1 -- 172.21.15.41:0/4150881304 <== mon.1 v1:172.21.15.41:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713591697 0 0) 0x7f1ab80a7e30 con 0x7f1ab805ca90 2021-03-09T08:54:15.078 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abc938700 1 -- 172.21.15.41:0/4150881304 --> v1:172.21.15.41:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa0003230 con 0x7f1ab805ca90 2021-03-09T08:54:15.078 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abe13b700 1 -- 172.21.15.41:0/4150881304 >> v2:172.21.15.38:3300/0 conn(0x7f1ab8099190 msgr2=0x7f1ab809bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:15.078 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.071+0000 7f1abe13b700 1 --2- 172.21.15.41:0/4150881304 >> v2:172.21.15.38:3300/0 conn(0x7f1ab8099190 0x7f1ab809bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.078 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abe13b700 1 -- 172.21.15.41:0/4150881304 >> v1:172.21.15.41:6789/0 conn(0x7f1ab805ca90 legacy=0x7f1ab80a7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.079 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abe13b700 1 -- 172.21.15.41:0/4150881304 --> v2:172.21.15.41:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80a82e0 con 0x7f1ab81398c0 2021-03-09T08:54:15.079 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 --2- 172.21.15.41:0/4150881304 >> v2:172.21.15.38:3300/0 conn(0x7f1ab8099190 0x7f1ab809bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-03-09T08:54:15.079 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abe13b700 1 --2- 172.21.15.41:0/4150881304 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab81398c0 0x7f1ab8139d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1ab80a9da0 tx=0x7f1ab00091d0).ready entity=mon.1 client_cookie=74c14cf2aa42159a server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.079 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abc938700 1 -- 172.21.15.41:0/4150881304 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000bad0 con 0x7f1ab81398c0 2021-03-09T08:54:15.079 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abc938700 1 -- 172.21.15.41:0/4150881304 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab000bc30 con 0x7f1ab81398c0 2021-03-09T08:54:15.080 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abc938700 1 -- 172.21.15.41:0/4150881304 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0011400 con 0x7f1ab81398c0 2021-03-09T08:54:15.080 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- 172.21.15.41:0/4150881304 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab81398c0 msgr2=0x7f1ab8139d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.080 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/4150881304 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab81398c0 0x7f1ab8139d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1ab80a9da0 tx=0x7f1ab00091d0).stop 2021-03-09T08:54:15.080 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- 172.21.15.41:0/4150881304 shutdown_connections 2021-03-09T08:54:15.081 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/4150881304 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab81398c0 0x7f1ab8139d50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.081 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/4150881304 >> v2:172.21.15.38:3300/0 conn(0x7f1ab8099190 0x7f1ab809bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.081 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- 172.21.15.41:0/4150881304 >> 172.21.15.41:0/4150881304 conn(0x7f1ab80649c0 msgr2=0x7f1ab8061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:15.081 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- 172.21.15.41:0/4150881304 shutdown_connections 2021-03-09T08:54:15.081 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- 172.21.15.41:0/4150881304 wait complete. 2021-03-09T08:54:15.082 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 Processor -- start 2021-03-09T08:54:15.082 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- start start 2021-03-09T08:54:15.085 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f1ab805ca90 0x7f1ab808f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.085 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 0x7f1ab808fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.085 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ab81398c0 0x7f1ab80a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.086 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80a5610 con 0x7f1ab81398c0 2021-03-09T08:54:15.086 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80a29a0 con 0x7f1ab8099190 2021-03-09T08:54:15.086 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abfb9e700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80a2c40 con 0x7f1ab805ca90 2021-03-09T08:54:15.086 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 0x7f1ab808fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.087 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 0x7f1ab808fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.41:3300/0 says I am v2:172.21.15.41:42248/0 (socket says 172.21.15.41:42248) 2021-03-09T08:54:15.087 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 -- 172.21.15.41:0/1201733671 learned_addr learned my addr 172.21.15.41:0/1201733671 (peer_addr_for_me v2:172.21.15.41:0/0) 2021-03-09T08:54:15.087 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abe13b700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ab81398c0 0x7f1ab80a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.087 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd93a700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f1ab805ca90 0x7f1ab808f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.088 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 -- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f1ab805ca90 msgr2=0x7f1ab808f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.088 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f1ab805ca90 0x7f1ab808f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.088 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 -- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ab81398c0 msgr2=0x7f1ab80a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.088 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ab81398c0 0x7f1ab80a5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.088 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 -- 172.21.15.41:0/1201733671 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80a4090 con 0x7f1ab8099190 2021-03-09T08:54:15.089 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.075+0000 7f1abd139700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 0x7f1ab808fbf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1aa800dcc0 tx=0x7f1aa8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.089 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800a870 con 0x7f1ab8099190 2021-03-09T08:54:15.089 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa800a9d0 con 0x7f1ab8099190 2021-03-09T08:54:15.089 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab809ba40 con 0x7f1ab8099190 2021-03-09T08:54:15.090 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800ad50 con 0x7f1ab8099190 2021-03-09T08:54:15.090 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab80a60d0 con 0x7f1ab8099190 2021-03-09T08:54:15.090 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f1aa8016030 con 0x7f1ab8099190 2021-03-09T08:54:15.090 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T08:54:15.090 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T08:54:15.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T08:54:15.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 4 mgrc reconnect No active mgr available yet 2021-03-09T08:54:15.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f1aa8005b70 con 0x7f1ab8099190 2021-03-09T08:54:15.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.079+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab80a6350 con 0x7f1ab8099190 2021-03-09T08:54:15.092 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.083+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f1aa801e060 con 0x7f1ab8099190 2021-03-09T08:54:15.166 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.167+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a847843b-12ff-4420-a4ea-c4c413008de2", "id": 1} v 0) v1 -- 0x7f1ab8045aa0 con 0x7f1ab8099190 2021-03-09T08:54:15.182 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1aa5ffb700 1 -- 172.21.15.41:0/1201733671 <== mon.1 v2:172.21.15.41:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a847843b-12ff-4420-a4ea-c4c413008de2", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1aa8014070 con 0x7f1ab8099190 2021-03-09T08:54:15.183 INFO:teuthology.orchestra.run.smithi041.stdout:1 2021-03-09T08:54:15.186 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 10 mgrc shutdown 2021-03-09T08:54:15.187 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 msgr2=0x7f1ab808fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.187 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 0x7f1ab808fbf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1aa800dcc0 tx=0x7f1aa8008040).stop 2021-03-09T08:54:15.187 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 shutdown_connections 2021-03-09T08:54:15.188 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ab81398c0 0x7f1ab80a5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.188 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f1ab805ca90 0x7f1ab808f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.189 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 --2- 172.21.15.41:0/1201733671 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f1ab8099190 0x7f1ab808fbf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.189 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 >> 172.21.15.41:0/1201733671 conn(0x7f1ab80649c0 msgr2=0x7f1ab80a76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:15.190 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 shutdown_connections 2021-03-09T08:54:15.190 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.183+0000 7f1abfb9e700 1 -- 172.21.15.41:0/1201733671 wait complete. 2021-03-09T08:54:15.202 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster ceph osd new 092fbe7e-df11-4410-aada-98a1dcb58200 2 2021-03-09T08:54:15.375 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 Processor -- start 2021-03-09T08:54:15.376 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 -- start start 2021-03-09T08:54:15.388 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f117c063360 0x7f117c099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.390 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f117c09a4a0 0x7f117c09b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.390 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 -- --> v1:172.21.15.38:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c063740 con 0x7f117c0a7040 2021-03-09T08:54:15.390 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 -- --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c09b010 con 0x7f117c063360 2021-03-09T08:54:15.391 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f11842f1700 1 -- --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c09beb0 con 0x7f117c09a4a0 2021-03-09T08:54:15.391 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f117c09a4a0 0x7f117c09b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.392 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f117c09a4a0 0x7f117c09b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.41:3300/0 says I am v2:172.21.15.41:42254/0 (socket says 172.21.15.41:42254) 2021-03-09T08:54:15.392 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 -- 172.21.15.41:0/3029262204 learned_addr learned my addr 172.21.15.41:0/3029262204 (peer_addr_for_me v2:172.21.15.41:0/0) 2021-03-09T08:54:15.393 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118208d700 1 --2- 172.21.15.41:0/3029262204 >> v2:172.21.15.38:3300/0 conn(0x7f117c063360 0x7f117c099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.393 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 -- 172.21.15.41:0/3029262204 >> v2:172.21.15.38:3300/0 conn(0x7f117c063360 msgr2=0x7f117c099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:15.394 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 --2- 172.21.15.41:0/3029262204 >> v2:172.21.15.38:3300/0 conn(0x7f117c063360 0x7f117c099fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.395 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 -- 172.21.15.41:0/3029262204 >> v1:172.21.15.38:6789/0 conn(0x7f117c0a7040 legacy=0x7f117c09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.395 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.375+0000 7f118188c700 1 -- 172.21.15.41:0/3029262204 --> v2:172.21.15.41:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c09acb0 con 0x7f117c09a4a0 2021-03-09T08:54:15.395 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f118188c700 1 --2- 172.21.15.41:0/3029262204 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c09b2b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f117c0a9c90 tx=0x7f116c006f30).ready entity=mon.1 client_cookie=b57bf97a66829442 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.396 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f118108b700 1 -- 172.21.15.41:0/3029262204 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00ca70 con 0x7f117c09a4a0 2021-03-09T08:54:15.396 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f118108b700 1 -- 172.21.15.41:0/3029262204 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c00cbd0 con 0x7f117c09a4a0 2021-03-09T08:54:15.397 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f118108b700 1 -- 172.21.15.41:0/3029262204 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00f470 con 0x7f117c09a4a0 2021-03-09T08:54:15.397 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 -- 172.21.15.41:0/3029262204 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 msgr2=0x7f117c09b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.398 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 --2- 172.21.15.41:0/3029262204 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c09b2b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f117c0a9c90 tx=0x7f116c006f30).stop 2021-03-09T08:54:15.398 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 -- 172.21.15.41:0/3029262204 shutdown_connections 2021-03-09T08:54:15.399 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 --2- 172.21.15.41:0/3029262204 >> v2:172.21.15.38:3300/0 conn(0x7f117c063360 0x7f117c099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.399 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 --2- 172.21.15.41:0/3029262204 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c09b2b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.399 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 -- 172.21.15.41:0/3029262204 >> 172.21.15.41:0/3029262204 conn(0x7f117c0649c0 msgr2=0x7f117c061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:15.400 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 -- 172.21.15.41:0/3029262204 shutdown_connections 2021-03-09T08:54:15.400 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 -- 172.21.15.41:0/3029262204 wait complete. 2021-03-09T08:54:15.401 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 Processor -- start 2021-03-09T08:54:15.401 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.379+0000 7f11842f1700 1 -- start start 2021-03-09T08:54:15.402 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f117c063360 0x7f117c0a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.402 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c065650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.402 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f117c0a7040 0x7f117c066330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.403 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0622a0 con 0x7f117c0a7040 2021-03-09T08:54:15.404 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c099ed0 con 0x7f117c09a4a0 2021-03-09T08:54:15.404 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0a3880 con 0x7f117c063360 2021-03-09T08:54:15.405 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c065650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.405 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c065650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.41:3300/0 says I am v2:172.21.15.41:42260/0 (socket says 172.21.15.41:42260) 2021-03-09T08:54:15.405 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 -- 172.21.15.41:0/1011284555 learned_addr learned my addr 172.21.15.41:0/1011284555 (peer_addr_for_me v2:172.21.15.41:0/0) 2021-03-09T08:54:15.406 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118208d700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f117c063360 0x7f117c0a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.406 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118288e700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f117c0a7040 0x7f117c066330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.407 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 -- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f117c063360 msgr2=0x7f117c0a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.407 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f117c063360 0x7f117c0a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.408 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 -- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f117c0a7040 msgr2=0x7f117c066330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.408 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f117c0a7040 0x7f117c066330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.408 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 -- 172.21.15.41:0/1011284555 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c066090 con 0x7f117c09a4a0 2021-03-09T08:54:15.409 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f118188c700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c065650 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f117c09bff0 tx=0x7f116c019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.409 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00f4b0 con 0x7f117c09a4a0 2021-03-09T08:54:15.410 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c00fe90 con 0x7f117c09a4a0 2021-03-09T08:54:15.410 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c0a3ce0 con 0x7f117c09a4a0 2021-03-09T08:54:15.411 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c0226b0 con 0x7f117c09a4a0 2021-03-09T08:54:15.411 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c0a2420 con 0x7f117c09a4a0 2021-03-09T08:54:15.412 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f116c02b070 con 0x7f117c09a4a0 2021-03-09T08:54:15.412 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T08:54:15.412 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T08:54:15.413 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T08:54:15.413 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 4 mgrc reconnect No active mgr available yet 2021-03-09T08:54:15.414 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f116c01d070 con 0x7f117c09a4a0 2021-03-09T08:54:15.414 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.383+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c0a2730 con 0x7f117c09a4a0 2021-03-09T08:54:15.415 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.387+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f116c00b040 con 0x7f117c09a4a0 2021-03-09T08:54:15.475 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.475+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "osd new", "uuid": "092fbe7e-df11-4410-aada-98a1dcb58200", "id": 2} v 0) v1 -- 0x7f117c045aa0 con 0x7f117c09a4a0 2021-03-09T08:54:15.496 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f116a7fc700 1 -- 172.21.15.41:0/1011284555 <== mon.1 v2:172.21.15.41:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "092fbe7e-df11-4410-aada-98a1dcb58200", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f116c0340b0 con 0x7f117c09a4a0 2021-03-09T08:54:15.497 INFO:teuthology.orchestra.run.smithi041.stdout:2 2021-03-09T08:54:15.500 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 10 mgrc shutdown 2021-03-09T08:54:15.500 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 msgr2=0x7f117c065650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.501 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c065650 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f117c09bff0 tx=0x7f116c019000).stop 2021-03-09T08:54:15.501 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 shutdown_connections 2021-03-09T08:54:15.502 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f117c0a7040 0x7f117c066330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.502 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f117c063360 0x7f117c0a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.502 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 --2- 172.21.15.41:0/1011284555 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f117c09a4a0 0x7f117c065650 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.503 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 >> 172.21.15.41:0/1011284555 conn(0x7f117c0649c0 msgr2=0x7f117c0a76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:15.503 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 shutdown_connections 2021-03-09T08:54:15.504 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.495+0000 7f11842f1700 1 -- 172.21.15.41:0/1011284555 wait complete. 2021-03-09T08:54:15.519 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster ceph osd new 3651e218-1140-4395-9dd9-dd8a74104986 3 2021-03-09T08:54:15.692 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c1ccac700 1 Processor -- start 2021-03-09T08:54:15.692 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c1ccac700 1 -- start start 2021-03-09T08:54:15.692 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c1ccac700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 0x7f2c1809d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.693 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c1ccac700 1 -- --> v1:172.21.15.38:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c180614a0 con 0x7f2c180610c0 2021-03-09T08:54:15.693 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c1ccac700 1 -- --> v1:172.21.15.41:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1809f8f0 con 0x7f2c180a9a60 2021-03-09T08:54:15.693 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c1ccac700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1809fb90 con 0x7f2c1809e1a0 2021-03-09T08:54:15.693 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c15d9b700 1 --1- >> v1:172.21.15.41:6789/0 conn(0x7f2c180a9a60 0x7f2c1813f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.41:6789/0 says I am v1:172.21.15.41:37470/0 (socket says 172.21.15.41:37470) 2021-03-09T08:54:15.694 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.683+0000 7f2c15d9b700 1 -- 172.21.15.41:0/1376186707 learned_addr learned my addr 172.21.15.41:0/1376186707 (peer_addr_for_me v1:172.21.15.41:0/0) 2021-03-09T08:54:15.694 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c16d9d700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 0x7f2c1809d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.694 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.1 v1:172.21.15.41:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913381977 0 0) 0x7f2c1809f8f0 con 0x7f2c180a9a60 2021-03-09T08:54:15.695 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 --> v1:172.21.15.41:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc003230 con 0x7f2c180a9a60 2021-03-09T08:54:15.695 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.0 v1:172.21.15.38:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233949825 0 0) 0x7f2c180614a0 con 0x7f2c180610c0 2021-03-09T08:54:15.695 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 --> v1:172.21.15.38:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c1809f8f0 con 0x7f2c180610c0 2021-03-09T08:54:15.695 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.1 v1:172.21.15.41:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3574181416 0 0) 0x7f2bfc003230 con 0x7f2c180a9a60 2021-03-09T08:54:15.696 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 msgr2=0x7f2c1809d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:15.696 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 0x7f2c1809d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.696 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 >> v1:172.21.15.38:6789/0 conn(0x7f2c180610c0 legacy=0x7f2c181365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.696 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 --> v1:172.21.15.41:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18136cd0 con 0x7f2c180a9a60 2021-03-09T08:54:15.696 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.1 v1:172.21.15.41:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 447678074 0 0) 0x7f2c0c0014e0 con 0x7f2c180a9a60 2021-03-09T08:54:15.697 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 >> v1:172.21.15.41:6789/0 conn(0x7f2c180a9a60 legacy=0x7f2c1813f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.697 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 0x7f2bfc0038b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.697 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2bfc003da0 0x7f2bfc0044d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.698 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bfc0049c0 0x7f2bfc004e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.698 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1809fb90 con 0x7f2bfc0049c0 2021-03-09T08:54:15.698 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc005330 con 0x7f2bfc003da0 2021-03-09T08:54:15.698 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc00b750 con 0x7f2c1809e1a0 2021-03-09T08:54:15.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2bfc003da0 0x7f2bfc0044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c16d9d700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bfc0049c0 0x7f2bfc004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1659c700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 0x7f2bfc0038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 -- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 msgr2=0x7f2bfc0038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:15.700 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c1809e1a0 0x7f2bfc0038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.700 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 -- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bfc0049c0 msgr2=0x7f2bfc004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:15.701 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bfc0049c0 0x7f2bfc004e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.701 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 -- 172.21.15.41:0/1376186707 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc003230 con 0x7f2bfc003da0 2021-03-09T08:54:15.701 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c16d9d700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bfc0049c0 0x7f2bfc004e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:54:15.702 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2bfc003da0 0x7f2bfc0044d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2c0c00c3f0 tx=0x7f2c0c007770).ready entity=mon.1 client_cookie=b1590b7d54a14de2 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.702 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c00ce50 con 0x7f2bfc003da0 2021-03-09T08:54:15.702 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c0070d0 con 0x7f2bfc003da0 2021-03-09T08:54:15.703 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1559a700 1 -- 172.21.15.41:0/1376186707 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c007470 con 0x7f2bfc003da0 2021-03-09T08:54:15.703 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 -- 172.21.15.41:0/1376186707 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2bfc003da0 msgr2=0x7f2bfc0044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.703 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 --2- 172.21.15.41:0/1376186707 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2bfc003da0 0x7f2bfc0044d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2c0c00c3f0 tx=0x7f2c0c007770).stop 2021-03-09T08:54:15.703 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c16d9d700 1 -- 172.21.15.41:0/1376186707 reap_dead start 2021-03-09T08:54:15.704 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 -- 172.21.15.41:0/1376186707 shutdown_connections 2021-03-09T08:54:15.704 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 -- 172.21.15.41:0/1376186707 >> 172.21.15.41:0/1376186707 conn(0x7f2c180628b0 msgr2=0x7f2c18064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:15.704 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 -- 172.21.15.41:0/1376186707 shutdown_connections 2021-03-09T08:54:15.704 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 -- 172.21.15.41:0/1376186707 wait complete. 2021-03-09T08:54:15.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 Processor -- start 2021-03-09T08:54:15.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.687+0000 7f2c1ccac700 1 -- start start 2021-03-09T08:54:15.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c180a6df0 0x7f2c180a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 0x7f2c181491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 0x7f2c18066fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180a5860 con 0x7f2c180a6df0 2021-03-09T08:54:15.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180a5b00 con 0x7f2c180a7d90 2021-03-09T08:54:15.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c181484e0 con 0x7f2c180a71d0 2021-03-09T08:54:15.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 0x7f2c181491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c16d9d700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 0x7f2c18066fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.708 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 0x7f2c181491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.41:3300/0 says I am v2:172.21.15.41:42278/0 (socket says 172.21.15.41:42278) 2021-03-09T08:54:15.708 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 -- 172.21.15.41:0/3360344364 learned_addr learned my addr 172.21.15.41:0/3360344364 (peer_addr_for_me v2:172.21.15.41:0/0) 2021-03-09T08:54:15.708 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c16d9d700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 0x7f2c18066fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3301/0 says I am v2:172.21.15.41:42892/0 (socket says 172.21.15.41:42892) 2021-03-09T08:54:15.708 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1659c700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c180a6df0 0x7f2c180a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.708 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 -- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 msgr2=0x7f2c18066fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.709 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 0x7f2c18066fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.709 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 -- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c180a6df0 msgr2=0x7f2c180a5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.709 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c180a6df0 0x7f2c180a5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.710 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 -- 172.21.15.41:0/3360344364 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18148c40 con 0x7f2c180a7d90 2021-03-09T08:54:15.710 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c16d9d700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 0x7f2c18066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:54:15.711 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c15d9b700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 0x7f2c181491c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2c0c00a000 tx=0x7f2c0c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.711 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c005a00 con 0x7f2c180a7d90 2021-03-09T08:54:15.712 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c005b60 con 0x7f2c180a7d90 2021-03-09T08:54:15.712 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c180a7630 con 0x7f2c180a7d90 2021-03-09T08:54:15.713 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c013450 con 0x7f2c180a7d90 2021-03-09T08:54:15.713 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18067640 con 0x7f2c180a7d90 2021-03-09T08:54:15.713 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c18067920 con 0x7f2c180a7d90 2021-03-09T08:54:15.714 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f2c0c01d070 con 0x7f2c180a7d90 2021-03-09T08:54:15.714 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T08:54:15.715 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T08:54:15.715 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T08:54:15.716 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 4 mgrc reconnect No active mgr available yet 2021-03-09T08:54:15.716 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.691+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f2c0c0011f0 con 0x7f2c180a7d90 2021-03-09T08:54:15.717 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.695+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f2c0c024020 con 0x7f2c180a7d90 2021-03-09T08:54:15.779 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.779+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3651e218-1140-4395-9dd9-dd8a74104986", "id": 3} v 0) v1 -- 0x7f2c18045aa0 con 0x7f2c180a7d90 2021-03-09T08:54:15.801 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.799+0000 7f2c067fc700 1 -- 172.21.15.41:0/3360344364 <== mon.1 v2:172.21.15.41:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3651e218-1140-4395-9dd9-dd8a74104986", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2c0c01a070 con 0x7f2c180a7d90 2021-03-09T08:54:15.802 INFO:teuthology.orchestra.run.smithi041.stdout:3 2021-03-09T08:54:15.803 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 10 mgrc shutdown 2021-03-09T08:54:15.804 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 msgr2=0x7f2c181491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:15.804 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 0x7f2c181491c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2c0c00a000 tx=0x7f2c0c015000).stop 2021-03-09T08:54:15.805 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 shutdown_connections 2021-03-09T08:54:15.805 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f2c180a71d0 0x7f2c18066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.805 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c180a6df0 0x7f2c180a5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.806 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 --2- 172.21.15.41:0/3360344364 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f2c180a7d90 0x7f2c181491c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.806 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 >> 172.21.15.41:0/3360344364 conn(0x7f2c180628b0 msgr2=0x7f2c18136390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:15.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 shutdown_connections 2021-03-09T08:54:15.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-03-09T08:54:15.803+0000 7f2c1ccac700 1 -- 172.21.15.41:0/3360344364 wait complete. 2021-03-09T08:54:15.823 INFO:tasks.ceph.osd.0:Restarting daemon 2021-03-09T08:54:15.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-03-09T08:54:15.826 INFO:tasks.ceph.osd.0:Started 2021-03-09T08:54:15.826 INFO:tasks.ceph.osd.1:Restarting daemon 2021-03-09T08:54:15.827 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-03-09T08:54:15.829 INFO:tasks.ceph.osd.1:Started 2021-03-09T08:54:15.830 INFO:tasks.ceph.osd.2:Restarting daemon 2021-03-09T08:54:15.830 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-03-09T08:54:15.832 INFO:tasks.ceph.osd.2:Started 2021-03-09T08:54:15.833 INFO:tasks.ceph.osd.3:Restarting daemon 2021-03-09T08:54:15.833 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-03-09T08:54:15.835 INFO:tasks.ceph.osd.3:Started 2021-03-09T08:54:15.836 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-09T08:54:16.240 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T08:54:16.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 Processor -- start 2021-03-09T08:54:16.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 -- start start 2021-03-09T08:54:16.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f60c40aac10 0x7f60c4061580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:16.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f60c4138d20 0x7f60c40a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:16.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 -- --> v1:172.21.15.38:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40aa1d0 con 0x7f60c40a9df0 2021-03-09T08:54:16.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 -- --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c41391d0 con 0x7f60c40aac10 2021-03-09T08:54:16.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60ca8a4700 1 -- --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c4062da0 con 0x7f60c4138d20 2021-03-09T08:54:16.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f60c40aac10 0x7f60c4061580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:16.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f60c40aac10 0x7f60c4061580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46500/0 (socket says 172.21.15.38:46500) 2021-03-09T08:54:16.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60c3fff700 1 --1- >> v1:172.21.15.38:6789/0 conn(0x7f60c40a9df0 0x7f60c4138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.38:6789/0 says I am v1:172.21.15.38:42864/0 (socket says 172.21.15.38:42864) 2021-03-09T08:54:16.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 -- 172.21.15.38:0/2581937274 learned_addr learned my addr 172.21.15.38:0/2581937274 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:16.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60c8e41700 1 --2- 172.21.15.38:0/2581937274 >> v2:172.21.15.41:3300/0 conn(0x7f60c4138d20 0x7f60c40a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:16.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 -- 172.21.15.38:0/2581937274 >> v2:172.21.15.41:3300/0 conn(0x7f60c4138d20 msgr2=0x7f60c40a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:16.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 --2- 172.21.15.38:0/2581937274 >> v2:172.21.15.41:3300/0 conn(0x7f60c4138d20 0x7f60c40a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 -- 172.21.15.38:0/2581937274 >> v1:172.21.15.38:6789/0 conn(0x7f60c40a9df0 legacy=0x7f60c4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:16.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60c37fe700 1 -- 172.21.15.38:0/2581937274 <== mon.0 v1:172.21.15.38:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682742871 0 0) 0x7f60c40aa1d0 con 0x7f60c40a9df0 2021-03-09T08:54:16.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60bbfff700 1 -- 172.21.15.38:0/2581937274 --> v2:172.21.15.38:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4061b30 con 0x7f60c40aac10 2021-03-09T08:54:16.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.398+0000 7f60c8e41700 1 --2- 172.21.15.38:0/2581937274 >> v2:172.21.15.41:3300/0 conn(0x7f60c4138d20 0x7f60c40a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-03-09T08:54:16.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60bbfff700 1 --2- 172.21.15.38:0/2581937274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c40aac10 0x7f60c4061580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f60c409d410 tx=0x7f60ac006ea0).ready entity=mon.0 client_cookie=9fd3ea802443fa22 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:16.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c37fe700 1 -- 172.21.15.38:0/2581937274 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac009920 con 0x7f60c40aac10 2021-03-09T08:54:16.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c37fe700 1 -- 172.21.15.38:0/2581937274 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac009a80 con 0x7f60c40aac10 2021-03-09T08:54:16.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c37fe700 1 -- 172.21.15.38:0/2581937274 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac009e00 con 0x7f60c40aac10 2021-03-09T08:54:16.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2581937274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c40aac10 msgr2=0x7f60c4061580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:16.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2581937274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c40aac10 0x7f60c4061580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f60c409d410 tx=0x7f60ac006ea0).stop 2021-03-09T08:54:16.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2581937274 shutdown_connections 2021-03-09T08:54:16.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2581937274 >> v2:172.21.15.41:3300/0 conn(0x7f60c4138d20 0x7f60c40a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2581937274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c40aac10 0x7f60c4061580 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2581937274 >> 172.21.15.38:0/2581937274 conn(0x7f60c4065f70 msgr2=0x7f60c4066380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:16.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2581937274 shutdown_connections 2021-03-09T08:54:16.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2581937274 wait complete. 2021-03-09T08:54:16.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 Processor -- start 2021-03-09T08:54:16.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- start start 2021-03-09T08:54:16.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f60c40a9df0 0x7f60c4140650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:16.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f60c40aac10 0x7f60c40a7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:16.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c4138d20 0x7f60c414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:16.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40a7600 con 0x7f60c4138d20 2021-03-09T08:54:16.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40a44f0 con 0x7f60c40aac10 2021-03-09T08:54:16.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40a5100 con 0x7f60c40a9df0 2021-03-09T08:54:16.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c3fff700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f60c40a9df0 0x7f60c4140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:16.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c4138d20 0x7f60c414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:16.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c3fff700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f60c40a9df0 0x7f60c4140650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3301/0 says I am v2:172.21.15.38:44182/0 (socket says 172.21.15.38:44182) 2021-03-09T08:54:16.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c3fff700 1 -- 172.21.15.38:0/2640677497 learned_addr learned my addr 172.21.15.38:0/2640677497 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:16.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60bbfff700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f60c40aac10 0x7f60c40a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:16.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 -- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f60c40a9df0 msgr2=0x7f60c4140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:16.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f60c40a9df0 0x7f60c4140650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 -- 172.21.15.38:0/2640677497 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f60c40aac10 msgr2=0x7f60c40a7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:16.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f60c40aac10 0x7f60c40a7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 -- 172.21.15.38:0/2640677497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40a3850 con 0x7f60c4138d20 2021-03-09T08:54:16.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c8e41700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c4138d20 0x7f60c414bf90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f60bc00db30 tx=0x7f60bc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:16.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00b7b0 con 0x7f60c4138d20 2021-03-09T08:54:16.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc00b910 con 0x7f60c4138d20 2021-03-09T08:54:16.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c414c480 con 0x7f60c4138d20 2021-03-09T08:54:16.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00bcb0 con 0x7f60c4138d20 2021-03-09T08:54:16.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c414c9a0 con 0x7f60c4138d20 2021-03-09T08:54:16.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f60bc005d20 con 0x7f60c4138d20 2021-03-09T08:54:16.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2021-03-09T08:54:16.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 4 mgrc handle_mgr_map Got map version 1 2021-03-09T08:54:16.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 4 mgrc handle_mgr_map Active mgr is now 2021-03-09T08:54:16.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 4 mgrc reconnect No active mgr available yet 2021-03-09T08:54:16.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.402+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f60bc012d60 con 0x7f60c4138d20 2021-03-09T08:54:16.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.406+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c40a39e0 con 0x7f60c4138d20 2021-03-09T08:54:16.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.406+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f60bc012630 con 0x7f60c4138d20 2021-03-09T08:54:16.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:54:16.424 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:54:16.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c414bcd0 con 0x7f60c4138d20 2021-03-09T08:54:16.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60c0ff9700 1 -- 172.21.15.38:0/2640677497 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4118 (secure 0 0 0) 0x7f60c414bcd0 con 0x7f60c4138d20 2021-03-09T08:54:16.486 INFO:teuthology.misc.health.smithi038.stdout: 2021-03-09T08:54:16.486 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":6,"fsid":"988e2677-23e9-442a-94d8-290838bc06dd","created":"2021-03-09T08:54:14.159633+0000","modified":"2021-03-09T08:54:15.781726+0000","last_up_change":"0.000000","last_in_change":"2021-03-09T08:54:15.781726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"dcfa5872-e85a-4aa3-a0c7-153e32b0fb29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a847843b-12ff-4420-a4ea-c4c413008de2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"092fbe7e-df11-4410-aada-98a1dcb58200","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3651e218-1140-4395-9dd9-dd8a74104986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T08:54:16.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 10 mgrc shutdown 2021-03-09T08:54:16.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c4138d20 msgr2=0x7f60c414bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:16.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c4138d20 0x7f60c414bf90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f60bc00db30 tx=0x7f60bc008040).stop 2021-03-09T08:54:16.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 shutdown_connections 2021-03-09T08:54:16.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60c4138d20 0x7f60c414bf90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f60c40a9df0 0x7f60c4140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 --2- 172.21.15.38:0/2640677497 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f60c40aac10 0x7f60c40a7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:16.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 >> 172.21.15.38:0/2640677497 conn(0x7f60c4065f70 msgr2=0x7f60c41383e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:16.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 shutdown_connections 2021-03-09T08:54:16.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:16.482+0000 7f60ca8a4700 1 -- 172.21.15.38:0/2640677497 wait complete. 2021-03-09T08:54:16.495 INFO:tasks.ceph.osd.2.smithi041.stderr:2021-03-09T08:54:16.491+0000 7f90449d1f00 -1 Falling back to public interface 2021-03-09T08:54:16.500 DEBUG:teuthology.misc:0 of 4 OSDs are up 2021-03-09T08:54:16.504 INFO:tasks.ceph.osd.0.smithi038.stderr:2021-03-09T08:54:16.502+0000 7fc637c6af00 -1 Falling back to public interface 2021-03-09T08:54:16.511 INFO:tasks.ceph.osd.3.smithi041.stderr:2021-03-09T08:54:16.511+0000 7fbe223ccf00 -1 Falling back to public interface 2021-03-09T08:54:16.514 INFO:tasks.ceph.osd.1.smithi041.stderr:2021-03-09T08:54:16.515+0000 7f0e1bd18f00 -1 Falling back to public interface 2021-03-09T08:54:20.147 INFO:tasks.ceph.osd.3.smithi041.stderr:2021-03-09T08:54:20.147+0000 7fbe223ccf00 -1 osd.3 0 log_to_monitors {default=true} 2021-03-09T08:54:20.247 INFO:tasks.ceph.osd.0.smithi038.stderr:2021-03-09T08:54:20.246+0000 7fc637c6af00 -1 osd.0 0 log_to_monitors {default=true} 2021-03-09T08:54:20.323 INFO:tasks.ceph.osd.2.smithi041.stderr:2021-03-09T08:54:20.323+0000 7f90449d1f00 -1 osd.2 0 log_to_monitors {default=true} 2021-03-09T08:54:20.348 INFO:tasks.ceph.osd.1.smithi041.stderr:2021-03-09T08:54:20.347+0000 7f0e1bd18f00 -1 osd.1 0 log_to_monitors {default=true} 2021-03-09T08:54:21.307 INFO:tasks.ceph.osd.0.smithi038.stderr:2021-03-09T08:54:21.302+0000 7fc62f134700 -1 osd.0 0 waiting for initial osdmap 2021-03-09T08:54:21.312 INFO:tasks.ceph.osd.3.smithi041.stderr:2021-03-09T08:54:21.311+0000 7fbe1b099700 -1 osd.3 0 waiting for initial osdmap 2021-03-09T08:54:22.377 INFO:tasks.ceph.mon.a.smithi038.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ff271e52700 time 2021-03-09T08:54:22.378895+0000 2021-03-09T08:54:22.378 INFO:tasks.ceph.mon.a.smithi038.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:22.385 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.385 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff27e6143dd] 2021-03-09T08:54:22.385 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.385 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.386 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.386 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.386 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.386 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.387 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.387 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.387 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.387 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.387 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.388 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.388 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.388 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.390 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: clone() 2021-03-09T08:54:22.390 INFO:tasks.ceph.mon.a.smithi038.stderr:*** Caught signal (Aborted) ** 2021-03-09T08:54:22.391 INFO:tasks.ceph.mon.a.smithi038.stderr: in thread 7ff271e52700 thread_name:ms_dispatch 2021-03-09T08:54:22.391 INFO:tasks.ceph.mon.a.smithi038.stderr:2021-03-09T08:54:22.382+0000 7ff271e52700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ff271e52700 time 2021-03-09T08:54:22.378895+0000 2021-03-09T08:54:22.391 INFO:tasks.ceph.mon.a.smithi038.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:22.394 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.395 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.395 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff27e6143dd] 2021-03-09T08:54:22.395 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.395 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.395 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.396 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.396 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.396 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.396 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.397 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.397 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.397 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.397 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.397 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.398 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.398 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.398 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: clone() 2021-03-09T08:54:22.398 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.399 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.400 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7ff27d48f980] 2021-03-09T08:54:22.400 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: gsignal() 2021-03-09T08:54:22.400 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: abort() 2021-03-09T08:54:22.400 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7ff27e61442e] 2021-03-09T08:54:22.400 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.401 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.401 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.401 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.401 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.402 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.402 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.402 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.402 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.403 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.403 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.403 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.403 INFO:tasks.ceph.mon.a.smithi038.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.403 INFO:tasks.ceph.mon.a.smithi038.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.404 INFO:tasks.ceph.mon.a.smithi038.stderr: 19: clone() 2021-03-09T08:54:22.404 INFO:tasks.ceph.mon.a.smithi038.stderr:2021-03-09T08:54:22.386+0000 7ff271e52700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:54:22.404 INFO:tasks.ceph.mon.a.smithi038.stderr: in thread 7ff271e52700 thread_name:ms_dispatch 2021-03-09T08:54:22.404 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.405 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.405 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7ff27d48f980] 2021-03-09T08:54:22.405 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: gsignal() 2021-03-09T08:54:22.405 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: abort() 2021-03-09T08:54:22.405 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7ff27e61442e] 2021-03-09T08:54:22.406 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.406 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.406 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.406 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.407 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.407 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.407 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.407 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.407 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.408 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.408 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.408 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.408 INFO:tasks.ceph.mon.a.smithi038.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.409 INFO:tasks.ceph.mon.a.smithi038.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.409 INFO:tasks.ceph.mon.a.smithi038.stderr: 19: clone() 2021-03-09T08:54:22.409 INFO:tasks.ceph.mon.a.smithi038.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:54:22.409 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.410 INFO:tasks.ceph.mon.a.smithi038.stderr: -1> 2021-03-09T08:54:22.382+0000 7ff271e52700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ff271e52700 time 2021-03-09T08:54:22.378895+0000 2021-03-09T08:54:22.410 INFO:tasks.ceph.mon.a.smithi038.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:22.411 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.411 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.411 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff27e6143dd] 2021-03-09T08:54:22.411 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.411 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.412 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.412 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.412 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.413 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.413 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.413 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.414 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.414 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.414 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.415 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.415 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.415 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.415 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: clone() 2021-03-09T08:54:22.416 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.416 INFO:tasks.ceph.mon.a.smithi038.stderr: 0> 2021-03-09T08:54:22.386+0000 7ff271e52700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:54:22.416 INFO:tasks.ceph.mon.a.smithi038.stderr: in thread 7ff271e52700 thread_name:ms_dispatch 2021-03-09T08:54:22.416 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.416 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.417 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7ff27d48f980] 2021-03-09T08:54:22.417 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: gsignal() 2021-03-09T08:54:22.417 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: abort() 2021-03-09T08:54:22.417 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7ff27e61442e] 2021-03-09T08:54:22.418 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.418 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.418 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.418 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.419 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.419 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.419 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.419 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.419 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.420 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.420 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.420 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.420 INFO:tasks.ceph.mon.a.smithi038.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.420 INFO:tasks.ceph.mon.a.smithi038.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.421 INFO:tasks.ceph.mon.a.smithi038.stderr: 19: clone() 2021-03-09T08:54:22.421 INFO:tasks.ceph.mon.a.smithi038.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:54:22.421 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.422 INFO:tasks.ceph.mon.a.smithi038.stderr: -1> 2021-03-09T08:54:22.382+0000 7ff271e52700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ff271e52700 time 2021-03-09T08:54:22.378895+0000 2021-03-09T08:54:22.422 INFO:tasks.ceph.mon.a.smithi038.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:22.423 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.423 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.423 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7ff27e6143dd] 2021-03-09T08:54:22.423 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.423 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.424 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.424 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.424 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.425 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.425 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.425 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.425 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.426 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.426 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.426 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.426 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.426 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.427 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: clone() 2021-03-09T08:54:22.427 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.427 INFO:tasks.ceph.mon.a.smithi038.stderr: 0> 2021-03-09T08:54:22.386+0000 7ff271e52700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:54:22.427 INFO:tasks.ceph.mon.a.smithi038.stderr: in thread 7ff271e52700 thread_name:ms_dispatch 2021-03-09T08:54:22.428 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.428 INFO:tasks.ceph.mon.a.smithi038.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:22.428 INFO:tasks.ceph.mon.a.smithi038.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7ff27d48f980] 2021-03-09T08:54:22.428 INFO:tasks.ceph.mon.a.smithi038.stderr: 2: gsignal() 2021-03-09T08:54:22.428 INFO:tasks.ceph.mon.a.smithi038.stderr: 3: abort() 2021-03-09T08:54:22.429 INFO:tasks.ceph.mon.a.smithi038.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7ff27e61442e] 2021-03-09T08:54:22.429 INFO:tasks.ceph.mon.a.smithi038.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7ff27e6145b8] 2021-03-09T08:54:22.429 INFO:tasks.ceph.mon.a.smithi038.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x5620bd779a65] 2021-03-09T08:54:22.429 INFO:tasks.ceph.mon.a.smithi038.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x5620bd7c5a71] 2021-03-09T08:54:22.429 INFO:tasks.ceph.mon.a.smithi038.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x5620bd7f5295] 2021-03-09T08:54:22.430 INFO:tasks.ceph.mon.a.smithi038.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x5620bd7fd24b] 2021-03-09T08:54:22.430 INFO:tasks.ceph.mon.a.smithi038.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x5620bd7fd7ab] 2021-03-09T08:54:22.430 INFO:tasks.ceph.mon.a.smithi038.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x5620bd76cade] 2021-03-09T08:54:22.430 INFO:tasks.ceph.mon.a.smithi038.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x5620bd657c79] 2021-03-09T08:54:22.431 INFO:tasks.ceph.mon.a.smithi038.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x5620bd65b229] 2021-03-09T08:54:22.431 INFO:tasks.ceph.mon.a.smithi038.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x5620bd65c66a] 2021-03-09T08:54:22.431 INFO:tasks.ceph.mon.a.smithi038.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5620bd68a178] 2021-03-09T08:54:22.431 INFO:tasks.ceph.mon.a.smithi038.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7ff27e8162ca] 2021-03-09T08:54:22.431 INFO:tasks.ceph.mon.a.smithi038.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff27e8c071d] 2021-03-09T08:54:22.432 INFO:tasks.ceph.mon.a.smithi038.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff27d4846db] 2021-03-09T08:54:22.432 INFO:tasks.ceph.mon.a.smithi038.stderr: 19: clone() 2021-03-09T08:54:22.432 INFO:tasks.ceph.mon.a.smithi038.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:54:22.432 INFO:tasks.ceph.mon.a.smithi038.stderr: 2021-03-09T08:54:22.905 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T08:54:23.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 Processor -- start 2021-03-09T08:54:23.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- start start 2021-03-09T08:54:23.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 0x7f8ee8149620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:23.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- --> v1:172.21.15.38:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8140b60 con 0x7f8ee8065aa0 2021-03-09T08:54:23.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- --> v1:172.21.15.41:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8140e00 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee813d3d0 con 0x7f8ee8065ee0 2021-03-09T08:54:23.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8ee7fff700 1 -- >> v1:172.21.15.38:6789/0 conn(0x7f8ee8065aa0 legacy=0x7f8ee8145ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.38:6789/0 2021-03-09T08:54:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 0x7f8ee8149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 0x7f8ee8149620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3301/0 says I am v2:172.21.15.38:44268/0 (socket says 172.21.15.38:44268) 2021-03-09T08:54:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 -- 172.21.15.38:0/127171752 learned_addr learned my addr 172.21.15.38:0/127171752 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 -- 172.21.15.38:0/127171752 >> v1:172.21.15.41:6789/0 conn(0x7f8ee80ab9d0 legacy=0x7f8ee8141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:54:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 -- 172.21.15.38:0/127171752 >> v1:172.21.15.38:6789/0 conn(0x7f8ee8065aa0 legacy=0x7f8ee8145ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 -- 172.21.15.38:0/127171752 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee8066370 con 0x7f8ee8065ee0 2021-03-09T08:54:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eed34f700 1 --2- 172.21.15.38:0/127171752 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 0x7f8ee8149620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8ee000ba10 tx=0x7f8ee0006ea0).ready entity=mon.2 client_cookie=2434e9e78afdbbdb server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:23.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8ee77fe700 1 -- 172.21.15.38:0/127171752 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee00098e0 con 0x7f8ee8065ee0 2021-03-09T08:54:23.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8ee77fe700 1 -- 172.21.15.38:0/127171752 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee0009a40 con 0x7f8ee8065ee0 2021-03-09T08:54:23.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8ee77fe700 1 -- 172.21.15.38:0/127171752 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0009dc0 con 0x7f8ee8065ee0 2021-03-09T08:54:23.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- 172.21.15.38:0/127171752 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 msgr2=0x7f8ee8149620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/127171752 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 0x7f8ee8149620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8ee000ba10 tx=0x7f8ee0006ea0).stop 2021-03-09T08:54:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- 172.21.15.38:0/127171752 shutdown_connections 2021-03-09T08:54:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/127171752 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee8065ee0 0x7f8ee8149620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:23.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- 172.21.15.38:0/127171752 >> 172.21.15.38:0/127171752 conn(0x7f8ee8062360 msgr2=0x7f8ee8062770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:23.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- 172.21.15.38:0/127171752 shutdown_connections 2021-03-09T08:54:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 -- 172.21.15.38:0/127171752 wait complete. 2021-03-09T08:54:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.082+0000 7f8eeedb2700 1 Processor -- start 2021-03-09T08:54:23.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 -- start start 2021-03-09T08:54:23.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ee8065aa0 0x7f8ee8099ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:23.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ee8065ee0 0x7f8ee814d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:23.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 0x7f8ee809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:23.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eecb4e700 1 -- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ee8065aa0 msgr2=0x7f8ee8099ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:54:23.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80a63b0 con 0x7f8ee8065aa0 2021-03-09T08:54:23.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eecb4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ee8065aa0 0x7f8ee8099ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:23.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee814cac0 con 0x7f8ee8065ee0 2021-03-09T08:54:23.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee809a640 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 0x7f8ee809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:23.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 0x7f8ee809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3301/0 says I am v2:172.21.15.38:44276/0 (socket says 172.21.15.38:44276) 2021-03-09T08:54:23.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8ee7fff700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ee8065ee0 0x7f8ee814d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:23.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 -- 172.21.15.38:0/140858501 learned_addr learned my addr 172.21.15.38:0/140858501 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:23.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 -- 172.21.15.38:0/140858501 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ee8065ee0 msgr2=0x7f8ee814d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:23.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ee8065ee0 0x7f8ee814d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:23.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 -- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ee8065aa0 msgr2=0x7f8ee8099ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:23.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ee8065aa0 0x7f8ee8099ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:23.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 -- 172.21.15.38:0/140858501 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee809a8e0 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8ee7fff700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ee8065ee0 0x7f8ee814d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:54:23.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eed34f700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 0x7f8ee809bed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f8ee0006ee0 tx=0x7f8ee0016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:23.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0020070 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee0009e80 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee80994f0 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee001c860 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.086+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee809c3c0 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.090+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee805e070 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.090+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 4 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f8ee000f070 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.090+0000 7f8ee4ff9700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:54:23.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8ee4ff9700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:54:23.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8ee4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:54:23.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8ee4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:54:23.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8ee4ff9700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f8ec8083100 0x7f8ec80855c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:23.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8eecb4e700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f8ec8083100 0x7f8ec80855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:23.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2212+0+0 (secure 0 0 0) 0x7f8ee00a6960 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.094+0000 7f8eecb4e700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f8ec8083100 0x7f8ec80855c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8edc0023b0 tx=0x7f8edc0027f0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:23.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.098+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7f8ee007bb40 con 0x7f8ee80ab9d0 2021-03-09T08:54:23.216 INFO:tasks.ceph.mon.a.smithi038.stderr:daemon-helper: command crashed with signal 6 2021-03-09T08:54:23.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:23.262+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ee805e070 con 0x7f8ee80ab9d0 2021-03-09T08:54:42.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.350+0000 7f8ee4ff9700 1 -- 172.21.15.38:0/140858501 <== mon.2 v2:172.21.15.38:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+4967 (secure 0 0 0) 0x7f8ee0083070 con 0x7f8ee80ab9d0 2021-03-09T08:54:42.353 INFO:teuthology.misc.health.smithi038.stdout: 2021-03-09T08:54:42.353 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":9,"fsid":"988e2677-23e9-442a-94d8-290838bc06dd","created":"2021-03-09T08:54:14.159633+0000","modified":"2021-03-09T08:54:22.302595+0000","last_up_change":"2021-03-09T08:54:22.302595+0000","last_in_change":"2021-03-09T08:54:15.781726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"dcfa5872-e85a-4aa3-a0c7-153e32b0fb29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":14516},{"type":"v1","addr":"172.21.15.38:6801","nonce":14516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":14516},{"type":"v1","addr":"172.21.15.38:6803","nonce":14516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":14516},{"type":"v1","addr":"172.21.15.38:6807","nonce":14516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":14516},{"type":"v1","addr":"172.21.15.38:6805","nonce":14516}]},"public_addr":"172.21.15.38:6801/14516","cluster_addr":"172.21.15.38:6803/14516","heartbeat_back_addr":"172.21.15.38:6807/14516","heartbeat_front_addr":"172.21.15.38:6805/14516","state":["exists","up"]},{"osd":1,"uuid":"a847843b-12ff-4420-a4ea-c4c413008de2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"092fbe7e-df11-4410-aada-98a1dcb58200","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3651e218-1140-4395-9dd9-dd8a74104986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6808","nonce":17008},{"type":"v1","addr":"172.21.15.41:6809","nonce":17008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6810","nonce":17008},{"type":"v1","addr":"172.21.15.41:6811","nonce":17008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6814","nonce":17008},{"type":"v1","addr":"172.21.15.41:6816","nonce":17008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6812","nonce":17008},{"type":"v1","addr":"172.21.15.41:6813","nonce":17008}]},"public_addr":"172.21.15.41:6809/17008","cluster_addr":"172.21.15.41:6811/17008","heartbeat_back_addr":"172.21.15.41:6816/17008","heartbeat_front_addr":"172.21.15.41:6813/17008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T08:54:42.354 INFO:tasks.ceph.osd.2.smithi041.stderr:2021-03-09T08:54:42.351+0000 7f903d69e700 -1 osd.2 0 waiting for initial osdmap 2021-03-09T08:54:42.354 INFO:tasks.ceph.osd.1.smithi041.stderr:2021-03-09T08:54:42.355+0000 7f0e131e2700 -1 osd.1 0 waiting for initial osdmap 2021-03-09T08:54:42.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 10 mgrc shutdown 2021-03-09T08:54:42.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f8ec8083100 msgr2=0x7f8ec80855c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:42.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f8ec8083100 0x7f8ec80855c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8edc0023b0 tx=0x7f8edc0027f0).stop 2021-03-09T08:54:42.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 msgr2=0x7f8ee809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:42.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 0x7f8ee809bed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f8ee0006ee0 tx=0x7f8ee0016000).stop 2021-03-09T08:54:42.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 shutdown_connections 2021-03-09T08:54:42.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f8ec8083100 0x7f8ec80855c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:42.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f8ee80ab9d0 0x7f8ee809bed0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:42.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ee8065aa0 0x7f8ee8099ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:42.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 --2- 172.21.15.38:0/140858501 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f8ee8065ee0 0x7f8ee814d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:42.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 >> 172.21.15.38:0/140858501 conn(0x7f8ee8062360 msgr2=0x7f8ee80631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:42.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 shutdown_connections 2021-03-09T08:54:42.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:42.354+0000 7f8eeedb2700 1 -- 172.21.15.38:0/140858501 wait complete. 2021-03-09T08:54:42.379 DEBUG:teuthology.misc:2 of 4 OSDs are up 2021-03-09T08:54:42.422 INFO:tasks.ceph.mon.b.smithi041.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb781ce2700 time 2021-03-09T08:54:42.425567+0000 2021-03-09T08:54:42.423 INFO:tasks.ceph.mon.b.smithi041.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:42.437 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.438 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fb78fca73dd] 2021-03-09T08:54:42.438 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.438 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.439 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.439 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.439 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.439 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.440 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.440 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.440 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.440 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.440 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.441 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.441 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.441 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.441 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.442 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.442 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.442 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.442 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.442 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: clone() 2021-03-09T08:54:42.443 INFO:tasks.ceph.mon.b.smithi041.stderr:*** Caught signal (Aborted) ** 2021-03-09T08:54:42.443 INFO:tasks.ceph.mon.b.smithi041.stderr: in thread 7fb781ce2700 thread_name:fn_monstore 2021-03-09T08:54:42.443 INFO:tasks.ceph.mon.b.smithi041.stderr:2021-03-09T08:54:42.439+0000 7fb781ce2700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb781ce2700 time 2021-03-09T08:54:42.425567+0000 2021-03-09T08:54:42.443 INFO:tasks.ceph.mon.b.smithi041.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:42.444 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.444 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.444 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fb78fca73dd] 2021-03-09T08:54:42.444 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.444 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.445 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.445 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.445 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.445 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.446 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.446 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.447 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.447 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.447 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.448 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.448 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.448 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.448 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.449 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.449 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.450 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.450 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.450 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: clone() 2021-03-09T08:54:42.451 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.452 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.452 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fb78eb22980] 2021-03-09T08:54:42.453 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: gsignal() 2021-03-09T08:54:42.453 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: abort() 2021-03-09T08:54:42.453 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fb78fca742e] 2021-03-09T08:54:42.453 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.454 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.454 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.454 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.455 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.455 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.455 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.456 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.456 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.457 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.457 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.458 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.458 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.458 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.458 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.459 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.459 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.459 INFO:tasks.ceph.mon.b.smithi041.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.459 INFO:tasks.ceph.mon.b.smithi041.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.460 INFO:tasks.ceph.mon.b.smithi041.stderr: 24: clone() 2021-03-09T08:54:42.460 INFO:tasks.ceph.mon.b.smithi041.stderr:2021-03-09T08:54:42.443+0000 7fb781ce2700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:54:42.460 INFO:tasks.ceph.mon.b.smithi041.stderr: in thread 7fb781ce2700 thread_name:fn_monstore 2021-03-09T08:54:42.460 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.460 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.461 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fb78eb22980] 2021-03-09T08:54:42.461 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: gsignal() 2021-03-09T08:54:42.461 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: abort() 2021-03-09T08:54:42.461 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fb78fca742e] 2021-03-09T08:54:42.462 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.462 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.462 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.462 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.463 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.463 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.463 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.463 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.463 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.464 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.464 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.464 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.464 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.465 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.465 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.465 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.465 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.465 INFO:tasks.ceph.mon.b.smithi041.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.466 INFO:tasks.ceph.mon.b.smithi041.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.466 INFO:tasks.ceph.mon.b.smithi041.stderr: 24: clone() 2021-03-09T08:54:42.466 INFO:tasks.ceph.mon.b.smithi041.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:54:42.466 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.472 INFO:tasks.ceph.mon.b.smithi041.stderr: -1> 2021-03-09T08:54:42.439+0000 7fb781ce2700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb781ce2700 time 2021-03-09T08:54:42.425567+0000 2021-03-09T08:54:42.473 INFO:tasks.ceph.mon.b.smithi041.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:42.473 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.473 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.474 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fb78fca73dd] 2021-03-09T08:54:42.474 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.474 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.474 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.475 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.475 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.475 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.475 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.475 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.476 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.476 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.476 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.476 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.477 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.477 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.477 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.477 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.478 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.478 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.478 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.478 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: clone() 2021-03-09T08:54:42.478 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.479 INFO:tasks.ceph.mon.b.smithi041.stderr: 0> 2021-03-09T08:54:42.443+0000 7fb781ce2700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:54:42.479 INFO:tasks.ceph.mon.b.smithi041.stderr: in thread 7fb781ce2700 thread_name:fn_monstore 2021-03-09T08:54:42.479 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.479 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.480 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fb78eb22980] 2021-03-09T08:54:42.480 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: gsignal() 2021-03-09T08:54:42.480 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: abort() 2021-03-09T08:54:42.480 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fb78fca742e] 2021-03-09T08:54:42.480 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.481 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.481 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.481 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.482 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.482 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.482 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.482 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.483 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.483 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.483 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.484 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.484 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.484 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.484 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.485 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.485 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.485 INFO:tasks.ceph.mon.b.smithi041.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.485 INFO:tasks.ceph.mon.b.smithi041.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.485 INFO:tasks.ceph.mon.b.smithi041.stderr: 24: clone() 2021-03-09T08:54:42.486 INFO:tasks.ceph.mon.b.smithi041.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:54:42.486 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.486 INFO:tasks.ceph.mon.b.smithi041.stderr: -1> 2021-03-09T08:54:42.439+0000 7fb781ce2700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb781ce2700 time 2021-03-09T08:54:42.425567+0000 2021-03-09T08:54:42.486 INFO:tasks.ceph.mon.b.smithi041.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:54:42.487 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.487 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.487 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fb78fca73dd] 2021-03-09T08:54:42.487 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.488 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.488 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.488 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.488 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.489 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.489 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.489 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.489 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.489 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.490 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.490 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.490 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.490 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.491 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.492 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.493 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.493 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.494 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.494 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: clone() 2021-03-09T08:54:42.494 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.494 INFO:tasks.ceph.mon.b.smithi041.stderr: 0> 2021-03-09T08:54:42.443+0000 7fb781ce2700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:54:42.494 INFO:tasks.ceph.mon.b.smithi041.stderr: in thread 7fb781ce2700 thread_name:fn_monstore 2021-03-09T08:54:42.495 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:42.495 INFO:tasks.ceph.mon.b.smithi041.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:54:42.495 INFO:tasks.ceph.mon.b.smithi041.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fb78eb22980] 2021-03-09T08:54:42.495 INFO:tasks.ceph.mon.b.smithi041.stderr: 2: gsignal() 2021-03-09T08:54:42.496 INFO:tasks.ceph.mon.b.smithi041.stderr: 3: abort() 2021-03-09T08:54:42.496 INFO:tasks.ceph.mon.b.smithi041.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fb78fca742e] 2021-03-09T08:54:42.496 INFO:tasks.ceph.mon.b.smithi041.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb78fca75b8] 2021-03-09T08:54:42.496 INFO:tasks.ceph.mon.b.smithi041.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x562e1ff83a65] 2021-03-09T08:54:42.496 INFO:tasks.ceph.mon.b.smithi041.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x562e1ffcfa71] 2021-03-09T08:54:42.497 INFO:tasks.ceph.mon.b.smithi041.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x562e1ffff295] 2021-03-09T08:54:42.497 INFO:tasks.ceph.mon.b.smithi041.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x562e2000724b] 2021-03-09T08:54:42.497 INFO:tasks.ceph.mon.b.smithi041.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x562e200077ab] 2021-03-09T08:54:42.498 INFO:tasks.ceph.mon.b.smithi041.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x562e1ff76ade] 2021-03-09T08:54:42.498 INFO:tasks.ceph.mon.b.smithi041.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x562e1fecc134] 2021-03-09T08:54:42.498 INFO:tasks.ceph.mon.b.smithi041.stderr: 13: (C_MonOp::finish(int)+0x45) [0x562e1fe6aaa5] 2021-03-09T08:54:42.498 INFO:tasks.ceph.mon.b.smithi041.stderr: 14: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.499 INFO:tasks.ceph.mon.b.smithi041.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x562e1fe93548] 2021-03-09T08:54:42.499 INFO:tasks.ceph.mon.b.smithi041.stderr: 16: (Paxos::finish_round()+0x9b) [0x562e1ff6e7db] 2021-03-09T08:54:42.499 INFO:tasks.ceph.mon.b.smithi041.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x562e1ff710c0] 2021-03-09T08:54:42.499 INFO:tasks.ceph.mon.b.smithi041.stderr: 18: (C_Committed::finish(int)+0x41) [0x562e1ff74861] 2021-03-09T08:54:42.500 INFO:tasks.ceph.mon.b.smithi041.stderr: 19: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.500 INFO:tasks.ceph.mon.b.smithi041.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x562e1ff745bf] 2021-03-09T08:54:42.500 INFO:tasks.ceph.mon.b.smithi041.stderr: 21: (Context::complete(int)+0x9) [0x562e1fe68059] 2021-03-09T08:54:42.500 INFO:tasks.ceph.mon.b.smithi041.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fb78fd083f5] 2021-03-09T08:54:42.500 INFO:tasks.ceph.mon.b.smithi041.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb78eb176db] 2021-03-09T08:54:42.501 INFO:tasks.ceph.mon.b.smithi041.stderr: 24: clone() 2021-03-09T08:54:42.501 INFO:tasks.ceph.mon.b.smithi041.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:54:42.501 INFO:tasks.ceph.mon.b.smithi041.stderr: 2021-03-09T08:54:43.304 INFO:tasks.ceph.mon.b.smithi041.stderr:daemon-helper: command crashed with signal 6 2021-03-09T08:54:46.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:54:46.517 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:54:48.784 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T08:54:48.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f428d700 1 Processor -- start 2021-03-09T08:54:48.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f428d700 1 -- start start 2021-03-09T08:54:48.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f428d700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 0x7f21ec149470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:48.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f428d700 1 -- --> v1:172.21.15.38:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ec13d340 con 0x7f21ec0625e0 2021-03-09T08:54:48.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f428d700 1 -- --> v1:172.21.15.41:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ec0629c0 con 0x7f21ec0ac060 2021-03-09T08:54:48.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f428d700 1 -- --> v2:172.21.15.41:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ec0ac4a0 con 0x7f21ec13cef0 2021-03-09T08:54:48.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f2029700 1 -- >> v1:172.21.15.38:6789/0 conn(0x7f21ec0625e0 legacy=0x7f21ec13c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.38:6789/0 2021-03-09T08:54:48.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f1828700 1 -- >> v1:172.21.15.41:6789/0 conn(0x7f21ec0ac060 legacy=0x7f21ec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.41:6789/0 2021-03-09T08:54:48.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f282a700 1 -- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 msgr2=0x7f21ec149470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:54:48.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:48.962+0000 7f21f282a700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 0x7f21ec149470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:49.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.166+0000 7f21f2029700 1 -- >> v1:172.21.15.38:6789/0 conn(0x7f21ec0625e0 legacy=0x7f21ec13c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.38:6789/0 2021-03-09T08:54:49.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.166+0000 7f21f1828700 1 -- >> v1:172.21.15.41:6789/0 conn(0x7f21ec0ac060 legacy=0x7f21ec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.41:6789/0 2021-03-09T08:54:49.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.166+0000 7f21f282a700 1 -- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 msgr2=0x7f21ec149470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:54:49.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.166+0000 7f21f282a700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 0x7f21ec149470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:54:49.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.570+0000 7f21f2029700 1 -- >> v1:172.21.15.38:6789/0 conn(0x7f21ec0625e0 legacy=0x7f21ec13c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.38:6789/0 2021-03-09T08:54:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.570+0000 7f21f1828700 1 -- >> v1:172.21.15.41:6789/0 conn(0x7f21ec0ac060 legacy=0x7f21ec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.41:6789/0 2021-03-09T08:54:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.570+0000 7f21f282a700 1 -- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 msgr2=0x7f21ec149470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:54:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:49.570+0000 7f21f282a700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 0x7f21ec149470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:54:50.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:50.374+0000 7f21f2029700 1 -- >> v1:172.21.15.38:6789/0 conn(0x7f21ec0625e0 legacy=0x7f21ec13c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.38:6789/0 2021-03-09T08:54:50.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:50.374+0000 7f21f1828700 1 -- >> v1:172.21.15.41:6789/0 conn(0x7f21ec0ac060 legacy=0x7f21ec145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.41:6789/0 2021-03-09T08:54:50.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:50.374+0000 7f21f282a700 1 -- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 msgr2=0x7f21ec149470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:54:50.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:50.374+0000 7f21f282a700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 0x7f21ec149470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:54:51.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 -- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 msgr2=0x7f21ec149470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:51.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 --2- >> v2:172.21.15.41:3300/0 conn(0x7f21ec13cef0 0x7f21ec149470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:51.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 -- >> v1:172.21.15.41:6789/0 conn(0x7f21ec0ac060 legacy=0x7f21ec145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:51.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 -- >> v1:172.21.15.38:6789/0 conn(0x7f21ec0625e0 legacy=0x7f21ec13c7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:51.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f21cc003380 0x7f21cc005800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:51.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc005e30 0x7f21cc0082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:51.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 -- --> v2:172.21.15.38:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ec0ac4a0 con 0x7f21cc003380 2021-03-09T08:54:51.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21ebfff700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008920 con 0x7f21cc005e30 2021-03-09T08:54:51.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f2029700 1 -- >> v2:172.21.15.38:3300/0 conn(0x7f21cc003380 msgr2=0x7f21cc005800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:54:51.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f2029700 1 --2- >> v2:172.21.15.38:3300/0 conn(0x7f21cc003380 0x7f21cc005800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:51.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc005e30 0x7f21cc0082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:51.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc005e30 0x7f21cc0082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3301/0 says I am v2:172.21.15.38:44344/0 (socket says 172.21.15.38:44344) 2021-03-09T08:54:51.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 -- 172.21.15.38:0/1612996938 learned_addr learned my addr 172.21.15.38:0/1612996938 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:51.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 -- 172.21.15.38:0/1612996938 >> v2:172.21.15.38:3300/0 conn(0x7f21cc003380 msgr2=0x7f21cc005800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:54:51.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 --2- 172.21.15.38:0/1612996938 >> v2:172.21.15.38:3300/0 conn(0x7f21cc003380 0x7f21cc005800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:51.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 -- 172.21.15.38:0/1612996938 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec0aaf70 con 0x7f21cc005e30 2021-03-09T08:54:51.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1828700 1 --2- 172.21.15.38:0/1612996938 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc005e30 0x7f21cc0082f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f21e0009a60 tx=0x7f21e0006f90).ready entity=mon.2 client_cookie=4fffe83093cb482d server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:51.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1027700 1 -- 172.21.15.38:0/1612996938 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0005320 con 0x7f21cc005e30 2021-03-09T08:54:51.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1027700 1 -- 172.21.15.38:0/1612996938 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0005480 con 0x7f21cc005e30 2021-03-09T08:54:51.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.962+0000 7f21f1027700 1 -- 172.21.15.38:0/1612996938 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003850 con 0x7f21cc005e30 2021-03-09T08:54:51.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 -- 172.21.15.38:0/1612996938 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:54:51.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 --2- 172.21.15.38:0/1612996938 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc005e30 0x7f21cc0082f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f21e0009a60 tx=0x7f21e0006f90).stop 2021-03-09T08:54:51.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f282a700 1 -- 172.21.15.38:0/1612996938 reap_dead start 2021-03-09T08:54:51.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 -- 172.21.15.38:0/1612996938 shutdown_connections 2021-03-09T08:54:51.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 -- 172.21.15.38:0/1612996938 >> 172.21.15.38:0/1612996938 conn(0x7f21ec0ad190 msgr2=0x7f21ec060d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:54:51.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 -- 172.21.15.38:0/1612996938 shutdown_connections 2021-03-09T08:54:51.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 -- 172.21.15.38:0/1612996938 wait complete. 2021-03-09T08:54:51.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 Processor -- start 2021-03-09T08:54:51.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.966+0000 7f21f428d700 1 -- start start 2021-03-09T08:54:51.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21ec13f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:51.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:51.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21ec097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:51.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0a98a0 con 0x7f21cc005e30 2021-03-09T08:54:51.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 -- --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0994e0 con 0x7f21ec09c280 2021-03-09T08:54:51.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 -- --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec09a490 con 0x7f21ec0a7f50 2021-03-09T08:54:51.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f2029700 1 -- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21ec13f280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:54:51.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f2029700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21ec13f280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:51.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:51.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f282a700 1 -- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21ec097c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:54:51.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 --2- >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3301/0 says I am v2:172.21.15.38:44346/0 (socket says 172.21.15.38:44346) 2021-03-09T08:54:51.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f282a700 1 --2- >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21ec097c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:51.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 learned_addr learned my addr 172.21.15.38:0/3657190860 (peer_addr_for_me v2:172.21.15.38:0/0) 2021-03-09T08:54:51.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21ec097c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:51.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21ec097c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:51.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21ec13f280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:51.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21ec13f280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:51.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec09a730 con 0x7f21ec0a7f50 2021-03-09T08:54:51.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f21e0003ef0 tx=0x7f21e0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:51.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000be20 con 0x7f21ec0a7f50 2021-03-09T08:54:51.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0012410 con 0x7f21ec0a7f50 2021-03-09T08:54:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec13fc90 con 0x7f21ec0a7f50 2021-03-09T08:54:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00127b0 con 0x7f21ec0a7f50 2021-03-09T08:54:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.970+0000 7f21f428d700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec0a83f0 con 0x7f21ec0a7f50 2021-03-09T08:54:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21f428d700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec045aa0 con 0x7f21ec0a7f50 2021-03-09T08:54:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 4 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e001a030 con 0x7f21ec0a7f50 2021-03-09T08:54:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:54:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:54:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:54:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:54:51.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0861f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:51.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2212+0+0 (secure 0 0 0) 0x7f21e00a73a0 con 0x7f21ec0a7f50 2021-03-09T08:54:51.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.974+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0861f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:51.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0861f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc006d70).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:51.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:51.982+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T08:54:52.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:54:52.146+0000 7f21f428d700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21ec045240 con 0x7f21ec0a7f50 2021-03-09T08:54:52.893 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:54:52.894+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:54:21.202464+0000) 2021-03-09T08:54:57.893 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:54:57.894+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:54:21.202464+0000) 2021-03-09T08:55:02.894 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:02.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21ec09bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:02.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21ec09bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:02.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f21e0003ef0 tx=0x7f21e0015000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:02.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f21e0003ef0 tx=0x7f21e0015000).stop 2021-03-09T08:55:02.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21ec09bd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:02.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21ec09bd90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:02.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc08a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:02.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:02.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:02.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08fac0 con 0x7f21cc005e30 2021-03-09T08:55:02.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08fc00 con 0x7f21ec09c280 2021-03-09T08:55:02.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08fd90 con 0x7f21cc0885c0 2021-03-09T08:55:02.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:02.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:02.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:02.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:02.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc08a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:02.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:02.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:02.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:02.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:02.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08ff20 con 0x7f21cc0885c0 2021-03-09T08:55:02.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21ec045aa0 con 0x7f21cc0885c0 2021-03-09T08:55:02.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc08a9d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f21e007fef0 tx=0x7f21e00844e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:02.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e008a070 con 0x7f21cc0885c0 2021-03-09T08:55:02.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c900 con 0x7f21cc0885c0 2021-03-09T08:55:02.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e000f030 con 0x7f21cc0885c0 2021-03-09T08:55:02.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.906+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:02.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:02.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:02.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:07.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc08a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:07.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc08a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:07.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc08a9d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f21e007fef0 tx=0x7f21e00844e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:07.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc08a9d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f21e007fef0 tx=0x7f21e00844e0).stop 2021-03-09T08:55:07.897 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:07.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc08a9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:07.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc08a9d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:07.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:07.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:07.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:07.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:07.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc001e70 con 0x7f21cc005e30 2021-03-09T08:55:07.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:07.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:07.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:07.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:07.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e050 con 0x7f21ec09c280 2021-03-09T08:55:07.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037430 con 0x7f21ec0a7f50 2021-03-09T08:55:07.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:07.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:07.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:07.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:07.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0312b0 con 0x7f21ec0a7f50 2021-03-09T08:55:07.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21ec0a7f50 2021-03-09T08:55:07.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f21dc015d60 tx=0x7f21dc015f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:07.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc013e60 con 0x7f21ec0a7f50 2021-03-09T08:55:07.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c040 con 0x7f21ec0a7f50 2021-03-09T08:55:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010400 con 0x7f21ec0a7f50 2021-03-09T08:55:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.902+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:07.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:12.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:14.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:14.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f21dc015d60 tx=0x7f21dc015f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:14.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f21dc015d60 tx=0x7f21dc015f70).stop 2021-03-09T08:55:14.171 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:14.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0921c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:14.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:14.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:14.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:14.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc054230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:14.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc005100 con 0x7f21cc005e30 2021-03-09T08:55:14.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc034000 con 0x7f21ec09c280 2021-03-09T08:55:14.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032af0 con 0x7f21cc0885c0 2021-03-09T08:55:14.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:14.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:14.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:14.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:14.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc054230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:14.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:14.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:14.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:14.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:14.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0085a0 con 0x7f21cc0885c0 2021-03-09T08:55:14.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21cc0885c0 2021-03-09T08:55:14.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc054230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f21dc07ffd0 tx=0x7f21dc0800a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:14.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07fc30 con 0x7f21cc0885c0 2021-03-09T08:55:14.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0844a0 con 0x7f21cc0885c0 2021-03-09T08:55:14.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07d9c0 con 0x7f21cc0885c0 2021-03-09T08:55:14.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.903+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:14.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:14.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:12.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:16.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:55:16.561 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:55:17.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc054230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:17.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc054230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:17.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc054230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f21dc07ffd0 tx=0x7f21dc0800a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:17.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc054230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f21dc07ffd0 tx=0x7f21dc0800a0).stop 2021-03-09T08:55:17.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc054230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:17.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc054230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:17.902 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:17.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:17.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:17.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:17.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0829a0 con 0x7f21cc005e30 2021-03-09T08:55:17.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc001d30 con 0x7f21ec09c280 2021-03-09T08:55:17.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ded0 con 0x7f21ec0a7f50 2021-03-09T08:55:17.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:17.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:17.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:17.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:17.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.894+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:17.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08d010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:17.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:17.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:17.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:17.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc055870 con 0x7f21ec0a7f50 2021-03-09T08:55:17.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:55:17.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f21ec0684b0 tx=0x7f21e400c850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:17.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400e040 con 0x7f21ec0a7f50 2021-03-09T08:55:17.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400e540 con 0x7f21ec0a7f50 2021-03-09T08:55:17.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4000f70 con 0x7f21ec0a7f50 2021-03-09T08:55:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.906+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:17.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:22.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:22.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:22.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f21ec0684b0 tx=0x7f21e400c850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f21ec0684b0 tx=0x7f21e400c850).stop 2021-03-09T08:55:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0921c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0921c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:22.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:22.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc0926b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:22.903 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:22.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051510 con 0x7f21cc005e30 2021-03-09T08:55:22.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051b40 con 0x7f21ec09c280 2021-03-09T08:55:22.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092ba0 con 0x7f21cc0885c0 2021-03-09T08:55:22.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:22.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:22.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:22.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:22.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc0926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:22.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:22.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:22.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:22.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:22.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092ce0 con 0x7f21cc0885c0 2021-03-09T08:55:22.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21cc0885c0 2021-03-09T08:55:22.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc0926b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f21dc07fbc0 tx=0x7f21dc088a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:22.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc079ea0 con 0x7f21cc0885c0 2021-03-09T08:55:22.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07dc50 con 0x7f21cc0885c0 2021-03-09T08:55:22.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc015620 con 0x7f21cc0885c0 2021-03-09T08:55:22.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.906+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:22.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:22.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:22.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:27.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc0926b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:55:27.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc0926b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:27.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc0926b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f21dc07fbc0 tx=0x7f21dc088a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:27.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc0926b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f21dc07fbc0 tx=0x7f21dc088a00).stop 2021-03-09T08:55:27.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc0926b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:27.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc0926b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:27.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.894+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:27.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:27.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:27.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0519f0 con 0x7f21cc005e30 2021-03-09T08:55:27.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02dd80 con 0x7f21ec09c280 2021-03-09T08:55:27.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:27.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0544f0 con 0x7f21ec0a7f50 2021-03-09T08:55:27.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:27.915 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:27.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:27.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:27.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:27.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:27.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:27.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:27.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:27.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023930 con 0x7f21ec0a7f50 2021-03-09T08:55:27.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21ec0a7f50 2021-03-09T08:55:27.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f21cc005d70 tx=0x7f21dc0800a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:27.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053610 con 0x7f21ec0a7f50 2021-03-09T08:55:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc053430 con 0x7f21ec0a7f50 2021-03-09T08:55:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc053a70 con 0x7f21ec0a7f50 2021-03-09T08:55:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.906+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:27.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:27.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:27.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:32.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc023440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:32.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc023440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:32.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f21cc005d70 tx=0x7f21dc0800a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:32.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f21cc005d70 tx=0x7f21dc0800a0).stop 2021-03-09T08:55:32.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc023440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:32.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023440 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:32.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:32.904 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 72 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc04ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08ad40 con 0x7f21cc005e30 2021-03-09T08:55:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0543a0 con 0x7f21ec09c280 2021-03-09T08:55:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032940 con 0x7f21cc0885c0 2021-03-09T08:55:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:32.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:32.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc04ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:32.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:32.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:32.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04bf40 con 0x7f21cc0885c0 2021-03-09T08:55:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21cc0885c0 2021-03-09T08:55:32.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc04ba50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f21cc05e9c0 tx=0x7f21dc077e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:32.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015450 con 0x7f21cc0885c0 2021-03-09T08:55:32.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015450 con 0x7f21cc0885c0 2021-03-09T08:55:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031190 con 0x7f21cc0885c0 2021-03-09T08:55:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:32.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:37.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc04ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:37.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc04ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:37.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc04ba50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f21cc05e9c0 tx=0x7f21dc077e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:37.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc04ba50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f21cc05e9c0 tx=0x7f21dc077e20).stop 2021-03-09T08:55:37.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 msgr2=0x7f21cc04ba50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:37.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0885c0 0x7f21cc04ba50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:37.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:37.905 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 106 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:37.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:37.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:37.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0826c0 con 0x7f21cc005e30 2021-03-09T08:55:37.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc07f0e0 con 0x7f21ec09c280 2021-03-09T08:55:37.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08ad20 con 0x7f21ec0a7f50 2021-03-09T08:55:37.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:37.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:37.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:37.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:37.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:37.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:37.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:37.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:37.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:37.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e910 con 0x7f21ec0a7f50 2021-03-09T08:55:37.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:55:37.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01e420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f21e4003b50 tx=0x7f21e407c890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:37.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T08:55:37.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400e330 con 0x7f21ec0a7f50 2021-03-09T08:55:37.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407c110 con 0x7f21ec0a7f50 2021-03-09T08:55:37.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.906+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:37.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:42.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:55:42.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:42.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01e420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f21e4003b50 tx=0x7f21e407c890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:42.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01e420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f21e4003b50 tx=0x7f21e407c890).stop 2021-03-09T08:55:42.905 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 136 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:42.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01e420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:42.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01e420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:42.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:42.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:42.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:42.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:42.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:42.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055730 con 0x7f21cc005e30 2021-03-09T08:55:42.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004fc0 con 0x7f21ec09c280 2021-03-09T08:55:42.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008440 con 0x7f21cc0457f0 2021-03-09T08:55:42.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:42.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:42.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.898+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:42.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:42.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:42.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:42.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:42.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047c80 con 0x7f21cc0457f0 2021-03-09T08:55:42.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21cc0457f0 2021-03-09T08:55:42.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f21dc009620 tx=0x7f21dc080110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:42.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc071b20 con 0x7f21cc0457f0 2021-03-09T08:55:42.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc071940 con 0x7f21cc0457f0 2021-03-09T08:55:42.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0457f0 2021-03-09T08:55:42.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:42.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:42.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:42.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:46.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:55:46.605 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:55:47.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc094e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:47.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc094e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:47.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f21dc009620 tx=0x7f21dc080110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:47.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f21dc009620 tx=0x7f21dc080110).stop 2021-03-09T08:55:47.906 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 171 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:47.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc094e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:47.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:47.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:47.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:47.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:47.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055730 con 0x7f21cc005e30 2021-03-09T08:55:47.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058670 con 0x7f21ec09c280 2021-03-09T08:55:47.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031060 con 0x7f21ec0a7f50 2021-03-09T08:55:47.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:47.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:47.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:47.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:47.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:47.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:47.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:47.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:47.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:47.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:55:47.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08ff20 con 0x7f21ec0a7f50 2021-03-09T08:55:47.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f21e007c6a0 tx=0x7f21e00a6020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:47.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21ec0a7f50 2021-03-09T08:55:47.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d710 con 0x7f21ec0a7f50 2021-03-09T08:55:47.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0059080 con 0x7f21ec0a7f50 2021-03-09T08:55:47.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:47.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:47.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:47.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:52.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:52.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:52.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f21e007c6a0 tx=0x7f21e00a6020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:52.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f21e007c6a0 tx=0x7f21e00a6020).stop 2021-03-09T08:55:52.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01a600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:52.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a600 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:52.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:52.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:52.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:52.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f500 con 0x7f21cc005e30 2021-03-09T08:55:52.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc07ce80 con 0x7f21ec09c280 2021-03-09T08:55:52.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031060 con 0x7f21cc0457f0 2021-03-09T08:55:52.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:52.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:52.910 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 205 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:52.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:52.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:52.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:52.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:52.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:52.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:52.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:52.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dd20 con 0x7f21cc0457f0 2021-03-09T08:55:52.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc015a40 con 0x7f21cc0457f0 2021-03-09T08:55:52.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f21dc06ef90 tx=0x7f21dc009520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:52.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a40 con 0x7f21cc0457f0 2021-03-09T08:55:52.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069390 con 0x7f21cc0457f0 2021-03-09T08:55:52.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0457f0 2021-03-09T08:55:52.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:52.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:52.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:52.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:55:57.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc094e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:57.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc094e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:57.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f21dc06ef90 tx=0x7f21dc009520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:57.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f21dc06ef90 tx=0x7f21dc009520).stop 2021-03-09T08:55:57.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc094e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:57.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc094e80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:57.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:57.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:57.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:57.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015d70 con 0x7f21cc005e30 2021-03-09T08:55:57.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dbd0 con 0x7f21ec09c280 2021-03-09T08:55:57.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055700 con 0x7f21ec0a7f50 2021-03-09T08:55:57.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:55:57.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:57.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:55:57.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:57.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:57.911 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 242 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:55:57.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:57.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:57.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:57.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:57.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc07ce80 con 0x7f21ec0a7f50 2021-03-09T08:55:57.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01dd20 con 0x7f21ec0a7f50 2021-03-09T08:55:57.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095370 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f21cc06f030 tx=0x7f21dc010fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:57.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0537b0 con 0x7f21ec0a7f50 2021-03-09T08:55:57.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015e10 con 0x7f21ec0a7f50 2021-03-09T08:55:57.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc068d40 con 0x7f21ec0a7f50 2021-03-09T08:55:57.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:55:57.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:55:57.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:55:57.910+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:02.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc095370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:56:02.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc095370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:02.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095370 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f21cc06f030 tx=0x7f21dc010fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:02.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095370 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f21cc06f030 tx=0x7f21dc010fd0).stop 2021-03-09T08:56:02.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc095370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:02.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095370 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:02.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:02.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:02.909 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 276 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:02.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:02.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc03e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:02.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:02.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:02.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dd70 con 0x7f21cc005e30 2021-03-09T08:56:02.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc076f30 con 0x7f21ec09c280 2021-03-09T08:56:02.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:02.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008460 con 0x7f21cc0457f0 2021-03-09T08:56:02.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc03e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:02.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:02.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:02.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:02.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:02.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08ab90 con 0x7f21cc0457f0 2021-03-09T08:56:02.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc07ce80 con 0x7f21cc0457f0 2021-03-09T08:56:02.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc03e200 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f21cc074e80 tx=0x7f21dc05f1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:02.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc066ba0 con 0x7f21cc0457f0 2021-03-09T08:56:02.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc066d00 con 0x7f21cc0457f0 2021-03-09T08:56:02.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05e870 con 0x7f21cc0457f0 2021-03-09T08:56:02.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:02.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:02.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:02.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:07.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc03e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:07.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc03e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:07.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc03e200 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f21cc074e80 tx=0x7f21dc05f1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:07.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc03e200 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f21cc074e80 tx=0x7f21dc05f1d0).stop 2021-03-09T08:56:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc03e200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc03e200 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:07.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:07.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:07.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc076f30 con 0x7f21cc005e30 2021-03-09T08:56:07.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc07c380 con 0x7f21ec09c280 2021-03-09T08:56:07.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ee30 con 0x7f21ec0a7f50 2021-03-09T08:56:07.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:07.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:07.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:07.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:07.913 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 314 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:07.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:07.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:07.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:07.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:07.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:07.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f4b0 con 0x7f21ec0a7f50 2021-03-09T08:56:07.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:56:07.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ef00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0006b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:07.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21ec0a7f50 2021-03-09T08:56:07.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007e330 con 0x7f21ec0a7f50 2021-03-09T08:56:07.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e005b490 con 0x7f21ec0a7f50 2021-03-09T08:56:07.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.910+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:07.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:07.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:07.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:12.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:12.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:12.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ef00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0006b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:12.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ef00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0006b90).stop 2021-03-09T08:56:12.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ef00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:12.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ef00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:12.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc029a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:12.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:12.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:12.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc076f30 con 0x7f21cc005e30 2021-03-09T08:56:12.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c820 con 0x7f21ec09c280 2021-03-09T08:56:12.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc07c360 con 0x7f21cc0457f0 2021-03-09T08:56:12.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:12.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:12.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:12.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:12.914 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 349 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:12.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc029a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:12.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:12.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:12.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:12.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:12.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0457f0 2021-03-09T08:56:12.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21cc0457f0 2021-03-09T08:56:12.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc029a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f21e0009030 tx=0x7f21e00314f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:12.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0084030 con 0x7f21cc0457f0 2021-03-09T08:56:12.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0084030 con 0x7f21cc0457f0 2021-03-09T08:56:12.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a2d80 con 0x7f21cc0457f0 2021-03-09T08:56:12.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.918+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:12.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:12.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:12.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:16.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:56:16.647 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:56:17.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc029a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:17.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc029a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:17.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc029a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f21e0009030 tx=0x7f21e00314f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc029a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f21e0009030 tx=0x7f21e00314f0).stop 2021-03-09T08:56:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc029a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc029a20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:17.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:17.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:17.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:17.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031100 con 0x7f21cc005e30 2021-03-09T08:56:17.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092a50 con 0x7f21ec09c280 2021-03-09T08:56:17.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc07c340 con 0x7f21ec0a7f50 2021-03-09T08:56:17.913 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 385 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:17.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:17.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:17.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:17.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:17.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:17.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:17.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:17.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:17.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:17.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc076f30 con 0x7f21ec0a7f50 2021-03-09T08:56:17.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:56:17.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f21e4056800 tx=0x7f21e40568e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:17.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21ec0a7f50 2021-03-09T08:56:17.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T08:56:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40098a0 con 0x7f21ec0a7f50 2021-03-09T08:56:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.918+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:17.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:17.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:22.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:22.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:22.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f21e4056800 tx=0x7f21e40568e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:22.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f21e4056800 tx=0x7f21e40568e0).stop 2021-03-09T08:56:22.912 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 421 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:22.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:22.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c580 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:22.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:22.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:22.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:22.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:22.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fd50 con 0x7f21cc005e30 2021-03-09T08:56:22.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:22.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:22.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f8f0 con 0x7f21ec09c280 2021-03-09T08:56:22.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f4e0 con 0x7f21cc0457f0 2021-03-09T08:56:22.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:22.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:22.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:22.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:22.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:22.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:22.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc037100 con 0x7f21cc0457f0 2021-03-09T08:56:22.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e400c310 con 0x7f21cc0457f0 2021-03-09T08:56:22.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0959e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f21e4030a90 tx=0x7f21e402d6f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:22.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40308d0 con 0x7f21cc0457f0 2021-03-09T08:56:22.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40306f0 con 0x7f21cc0457f0 2021-03-09T08:56:22.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40528f0 con 0x7f21cc0457f0 2021-03-09T08:56:22.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.918+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:22.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:22.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:22.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:27.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:56:27.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:27.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0959e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f21e4030a90 tx=0x7f21e402d6f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:27.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0959e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f21e4030a90 tx=0x7f21e402d6f0).stop 2021-03-09T08:56:27.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0959e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:27.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0959e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:27.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:27.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:27.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:27.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02dc40 con 0x7f21cc005e30 2021-03-09T08:56:27.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21ec09c280 2021-03-09T08:56:27.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055490 con 0x7f21ec0a7f50 2021-03-09T08:56:27.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:27.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:27.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:27.915 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 458 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:27.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:27.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:27.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:27.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:27.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:27.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:27.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b290 con 0x7f21ec0a7f50 2021-03-09T08:56:27.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08ab90 con 0x7f21ec0a7f50 2021-03-09T08:56:27.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ada0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f21cc071b50 tx=0x7f21dc009520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:27.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T08:56:27.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05efe0 con 0x7f21ec0a7f50 2021-03-09T08:56:27.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc084130 con 0x7f21ec0a7f50 2021-03-09T08:56:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.918+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:27.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:32.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:56:32.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:32.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ada0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f21cc071b50 tx=0x7f21dc009520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:32.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ada0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f21cc071b50 tx=0x7f21dc009520).stop 2021-03-09T08:56:32.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ada0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:32.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ada0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:32.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc027c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:32.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:32.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:32.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21cc005e30 2021-03-09T08:56:32.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03db50 con 0x7f21ec09c280 2021-03-09T08:56:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004fb0 con 0x7f21cc0457f0 2021-03-09T08:56:32.915 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 493 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:32.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:32.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:32.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:32.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:32.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc027c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:32.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:32.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:32.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:32.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:32.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc075460 con 0x7f21cc0457f0 2021-03-09T08:56:32.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0457f0 2021-03-09T08:56:32.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc027c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f21e0031820 tx=0x7f21e0006c70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:32.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a5e10 con 0x7f21cc0457f0 2021-03-09T08:56:32.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0457f0 2021-03-09T08:56:32.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009fda0 con 0x7f21cc0457f0 2021-03-09T08:56:32.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.918+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:32.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:32.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:32.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc027c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc027c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:37.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc027c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f21e0031820 tx=0x7f21e0006c70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:37.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc027c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f21e0031820 tx=0x7f21e0006c70).stop 2021-03-09T08:56:37.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc027c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:37.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc027c80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:37.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:37.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:37.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:37.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21cc005e30 2021-03-09T08:56:37.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045140 con 0x7f21ec09c280 2021-03-09T08:56:37.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02aad0 con 0x7f21ec0a7f50 2021-03-09T08:56:37.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:37.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:37.923 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 530 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:37.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:37.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:37.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:37.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:37.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:37.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:56:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b290 con 0x7f21ec0a7f50 2021-03-09T08:56:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0963a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f21dc031b50 tx=0x7f21dc070e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:37.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010df0 con 0x7f21ec0a7f50 2021-03-09T08:56:37.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010df0 con 0x7f21ec0a7f50 2021-03-09T08:56:37.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069080 con 0x7f21ec0a7f50 2021-03-09T08:56:37.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:37.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:37.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:37.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:42.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:56:42.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:42.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0963a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f21dc031b50 tx=0x7f21dc070e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:42.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0963a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f21dc031b50 tx=0x7f21dc070e50).stop 2021-03-09T08:56:42.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0963a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:42.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0963a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:42.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc043330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:42.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:42.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:42.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21cc005e30 2021-03-09T08:56:42.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052060 con 0x7f21ec09c280 2021-03-09T08:56:42.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058170 con 0x7f21cc0457f0 2021-03-09T08:56:42.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:42.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:42.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:42.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:42.918 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 564 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:42.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc043330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:42.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:42.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:42.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:42.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:42.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017ef0 con 0x7f21cc0457f0 2021-03-09T08:56:42.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc075460 con 0x7f21cc0457f0 2021-03-09T08:56:42.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc043330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00a61d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:42.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0099550 con 0x7f21cc0457f0 2021-03-09T08:56:42.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0099550 con 0x7f21cc0457f0 2021-03-09T08:56:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0095310 con 0x7f21cc0457f0 2021-03-09T08:56:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:42.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:46.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:56:46.698 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:56:47.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc043330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:56:47.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc043330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:47.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc043330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00a61d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:47.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc043330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00a61d0).stop 2021-03-09T08:56:47.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc043330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:47.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc043330 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:47.916 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 601 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:47.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:47.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:47.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:47.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21cc005e30 2021-03-09T08:56:47.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dad0 con 0x7f21ec09c280 2021-03-09T08:56:47.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055260 con 0x7f21ec0a7f50 2021-03-09T08:56:47.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:47.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:47.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:47.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:47.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:47.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:47.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:47.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:47.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:47.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04eba0 con 0x7f21ec0a7f50 2021-03-09T08:56:47.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017ef0 con 0x7f21ec0a7f50 2021-03-09T08:56:47.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02d520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f21e00a12e0 tx=0x7f21e00a1420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:47.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21ec0a7f50 2021-03-09T08:56:47.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21ec0a7f50 2021-03-09T08:56:47.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a0ad0 con 0x7f21ec0a7f50 2021-03-09T08:56:47.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:47.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:47.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:47.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:52.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:52.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:52.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02d520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f21e00a12e0 tx=0x7f21e00a1420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:52.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02d520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f21e00a12e0 tx=0x7f21e00a1420).stop 2021-03-09T08:56:52.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02d520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:52.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02d520 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:52.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:52.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:52.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:52.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21cc005e30 2021-03-09T08:56:52.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc06efc0 con 0x7f21ec09c280 2021-03-09T08:56:52.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0397b0 con 0x7f21cc0457f0 2021-03-09T08:56:52.934 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 636 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:52.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:52.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:52.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036f70 con 0x7f21cc0457f0 2021-03-09T08:56:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04eba0 con 0x7f21cc0457f0 2021-03-09T08:56:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0972e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0058b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0457f0 2021-03-09T08:56:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0038a30 con 0x7f21cc0457f0 2021-03-09T08:56:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0037090 con 0x7f21cc0457f0 2021-03-09T08:56:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:52.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:56:57.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:57.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:57.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0972e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0058b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:57.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0972e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0058b30).stop 2021-03-09T08:56:57.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0972e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:57.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0972e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:57.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:57.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:57.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:57.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0657b0 con 0x7f21cc005e30 2021-03-09T08:56:57.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc06efc0 con 0x7f21ec09c280 2021-03-09T08:56:57.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ceb0 con 0x7f21ec0a7f50 2021-03-09T08:56:57.920 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 674 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:56:57.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:56:57.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:57.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:56:57.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:57.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:57.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:57.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:57.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:57.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:57.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00c620 con 0x7f21ec0a7f50 2021-03-09T08:56:57.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:56:57.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f21cc06c700 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:57.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc031990 con 0x7f21ec0a7f50 2021-03-09T08:56:57.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21ec0a7f50 2021-03-09T08:56:57.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06e940 con 0x7f21ec0a7f50 2021-03-09T08:56:57.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:56:57.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:56:57.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:56:57.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:02.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:02.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:02.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f21cc06c700 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:02.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f21cc06c700 tx=0x7f21dc077de0).stop 2021-03-09T08:57:02.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:02.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012e30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:02.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:02.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:02.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:02.921 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 709 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:02.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc071bd0 con 0x7f21cc005e30 2021-03-09T08:57:02.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:02.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc060930 con 0x7f21ec09c280 2021-03-09T08:57:02.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc0457f0 2021-03-09T08:57:02.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:02.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:02.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:02.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0457f0 2021-03-09T08:57:02.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036f70 con 0x7f21cc0457f0 2021-03-09T08:57:02.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:02.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e005da40 con 0x7f21cc0457f0 2021-03-09T08:57:02.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e005d860 con 0x7f21cc0457f0 2021-03-09T08:57:02.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009e2f0 con 0x7f21cc0457f0 2021-03-09T08:57:02.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:02.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:02.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:02.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:07.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:07.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:07.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:07.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d670).stop 2021-03-09T08:57:07.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:07.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097140 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:07.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:07.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:07.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:07.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc072410 con 0x7f21cc005e30 2021-03-09T08:57:07.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc071bd0 con 0x7f21ec09c280 2021-03-09T08:57:07.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027f20 con 0x7f21ec0a7f50 2021-03-09T08:57:07.921 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 743 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:07.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:07.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:07.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:07.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:07.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:07.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:07.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:07.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:07.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:07.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc065760 con 0x7f21ec0a7f50 2021-03-09T08:57:07.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:57:07.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097630 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f21e40587e0 tx=0x7f21e4058990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:07.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21ec0a7f50 2021-03-09T08:57:07.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400c6b0 con 0x7f21ec0a7f50 2021-03-09T08:57:07.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400c6b0 con 0x7f21ec0a7f50 2021-03-09T08:57:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:07.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:12.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc097630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:12.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc097630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:12.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097630 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f21e40587e0 tx=0x7f21e4058990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:12.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097630 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f21e40587e0 tx=0x7f21e4058990).stop 2021-03-09T08:57:12.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc097630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:12.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097630 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:12.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:12.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:12.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:12.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c4e0 con 0x7f21cc005e30 2021-03-09T08:57:12.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0083a0 con 0x7f21ec09c280 2021-03-09T08:57:12.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02acd0 con 0x7f21cc0457f0 2021-03-09T08:57:12.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:12.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:12.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:12.922 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 780 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:12.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:12.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:12.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:12.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:12.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:12.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:12.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054360 con 0x7f21cc0457f0 2021-03-09T08:57:12.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00c620 con 0x7f21cc0457f0 2021-03-09T08:57:12.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097b20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f21cc03a490 tx=0x7f21dc070ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:12.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21cc0457f0 2021-03-09T08:57:12.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05e4f0 con 0x7f21cc0457f0 2021-03-09T08:57:12.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc084130 con 0x7f21cc0457f0 2021-03-09T08:57:12.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:12.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:12.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:12.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:16.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:57:16.749 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:57:17.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:17.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:17.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097b20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f21cc03a490 tx=0x7f21dc070ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:17.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097b20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f21cc03a490 tx=0x7f21dc070ec0).stop 2021-03-09T08:57:17.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:17.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097b20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:17.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0279c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc005e30 2021-03-09T08:57:17.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc06c640 con 0x7f21ec09c280 2021-03-09T08:57:17.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049880 con 0x7f21ec0a7f50 2021-03-09T08:57:17.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:17.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:17.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:17.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:17.923 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 815 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:17.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0279c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:17.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:17.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:17.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:17.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:17.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc072410 con 0x7f21ec0a7f50 2021-03-09T08:57:17.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054360 con 0x7f21ec0a7f50 2021-03-09T08:57:17.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0279c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f21cc045710 tx=0x7f21dc0331c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:17.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc084340 con 0x7f21ec0a7f50 2021-03-09T08:57:17.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc037510 con 0x7f21ec0a7f50 2021-03-09T08:57:17.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06e880 con 0x7f21ec0a7f50 2021-03-09T08:57:17.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:17.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:17.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:17.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:22.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0279c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:22.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0279c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:22.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0279c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f21cc045710 tx=0x7f21dc0331c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:22.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0279c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f21cc045710 tx=0x7f21dc0331c0).stop 2021-03-09T08:57:22.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0279c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:22.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0279c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:22.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc035730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:22.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:22.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:22.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc06c640 con 0x7f21cc005e30 2021-03-09T08:57:22.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec09c280 2021-03-09T08:57:22.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc075130 con 0x7f21cc0457f0 2021-03-09T08:57:22.926 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 852 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:22.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:22.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:22.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:22.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:22.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc035730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:22.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:22.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:22.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:22.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0637d0 con 0x7f21cc0457f0 2021-03-09T08:57:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0457f0 2021-03-09T08:57:22.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc035730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f21e000a010 tx=0x7f21e0006b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:22.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e005dea0 con 0x7f21cc0457f0 2021-03-09T08:57:22.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00850d0 con 0x7f21cc0457f0 2021-03-09T08:57:22.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0036260 con 0x7f21cc0457f0 2021-03-09T08:57:22.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:22.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:22.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:22.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:27.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc035730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc035730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc035730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f21e000a010 tx=0x7f21e0006b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc035730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f21e000a010 tx=0x7f21e0006b90).stop 2021-03-09T08:57:27.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc035730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:27.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc035730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:27.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:27.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:27.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:27.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc005e30 2021-03-09T08:57:27.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01df70 con 0x7f21ec09c280 2021-03-09T08:57:27.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022f90 con 0x7f21ec0a7f50 2021-03-09T08:57:27.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:27.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:27.925 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 888 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:27.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:27.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:27.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:27.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:27.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc06d590 con 0x7f21ec0a7f50 2021-03-09T08:57:27.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:57:27.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f21e40312f0 tx=0x7f21e40a31f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:27.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089020 con 0x7f21ec0a7f50 2021-03-09T08:57:27.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:57:27.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400c310 con 0x7f21ec0a7f50 2021-03-09T08:57:27.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.930+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:27.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:27.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:27.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:32.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:32.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:32.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f21e40312f0 tx=0x7f21e40a31f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:32.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f21e40312f0 tx=0x7f21e40a31f0).stop 2021-03-09T08:57:32.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:32.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:32.923 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 924 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:32.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:32.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:32.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:32.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049590 con 0x7f21cc005e30 2021-03-09T08:57:32.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01df70 con 0x7f21ec09c280 2021-03-09T08:57:32.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051d70 con 0x7f21cc0457f0 2021-03-09T08:57:32.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:32.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:32.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:32.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:32.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:32.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:32.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:32.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:32.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:32.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08ab90 con 0x7f21cc0457f0 2021-03-09T08:57:32.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc06d590 con 0x7f21cc0457f0 2021-03-09T08:57:32.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0978b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4003b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:32.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4032010 con 0x7f21cc0457f0 2021-03-09T08:57:32.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3030 con 0x7f21cc0457f0 2021-03-09T08:57:32.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402d0d0 con 0x7f21cc0457f0 2021-03-09T08:57:32.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:32.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:32.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:32.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:37.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:57:37.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:37.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0978b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4003b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:37.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0978b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4003b50).stop 2021-03-09T08:57:37.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc0978b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:37.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc0978b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:37.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:37.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:37.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:37.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022f90 con 0x7f21cc005e30 2021-03-09T08:57:37.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec09c280 2021-03-09T08:57:37.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01df70 con 0x7f21ec0a7f50 2021-03-09T08:57:37.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:37.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:37.926 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 961 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:37.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:37.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:57:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08ab90 con 0x7f21ec0a7f50 2021-03-09T08:57:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f21cc02d450 tx=0x7f21e4059b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:37.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402b050 con 0x7f21ec0a7f50 2021-03-09T08:57:37.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4031e00 con 0x7f21ec0a7f50 2021-03-09T08:57:37.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4059620 con 0x7f21ec0a7f50 2021-03-09T08:57:37.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.930+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:37.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:37.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:37.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:42.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f21cc02d450 tx=0x7f21e4059b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f21cc02d450 tx=0x7f21e4059b10).stop 2021-03-09T08:57:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc06d910 con 0x7f21cc005e30 2021-03-09T08:57:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045340 con 0x7f21ec09c280 2021-03-09T08:57:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc0457f0 2021-03-09T08:57:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:42.927 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:42.918+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 997 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:42.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:42.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:42.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:42.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:42.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:42.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:42.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:42.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:42.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00a610 con 0x7f21cc0457f0 2021-03-09T08:57:42.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc072410 con 0x7f21cc0457f0 2021-03-09T08:57:42.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097f30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:42.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc084340 con 0x7f21cc0457f0 2021-03-09T08:57:42.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06cd50 con 0x7f21cc0457f0 2021-03-09T08:57:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc0457f0 2021-03-09T08:57:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:42.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:46.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:57:46.795 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:57:47.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:47.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:47.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097f30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:47.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097f30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02df50).stop 2021-03-09T08:57:47.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc097f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:47.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc097f30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:47.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:47.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:47.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:47.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0327e0 con 0x7f21cc005e30 2021-03-09T08:57:47.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05d6a0 con 0x7f21ec09c280 2021-03-09T08:57:47.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc018140 con 0x7f21ec0a7f50 2021-03-09T08:57:47.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:47.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:47.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:47.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:47.928 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1032 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:47.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc045110 con 0x7f21ec0a7f50 2021-03-09T08:57:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00a610 con 0x7f21ec0a7f50 2021-03-09T08:57:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f21dc006e80 tx=0x7f21dc05c5c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:47.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05c000 con 0x7f21ec0a7f50 2021-03-09T08:57:47.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc064e90 con 0x7f21ec0a7f50 2021-03-09T08:57:47.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06abb0 con 0x7f21ec0a7f50 2021-03-09T08:57:47.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.930+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:47.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:52.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:52.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:52.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f21dc006e80 tx=0x7f21dc05c5c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:52.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f21dc006e80 tx=0x7f21dc05c5c0).stop 2021-03-09T08:57:52.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc098420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:52.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098420 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:52.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:52.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:52.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc098910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:52.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21cc005e30 2021-03-09T08:57:52.928 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1069 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:52.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040310 con 0x7f21ec09c280 2021-03-09T08:57:52.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022f90 con 0x7f21cc0457f0 2021-03-09T08:57:52.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:52.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:52.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:52.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:52.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:52.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:52.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:52.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:52.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:52.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098000 con 0x7f21cc0457f0 2021-03-09T08:57:52.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc045110 con 0x7f21cc0457f0 2021-03-09T08:57:52.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc098910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc064f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:52.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01fff0 con 0x7f21cc0457f0 2021-03-09T08:57:52.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05c000 con 0x7f21cc0457f0 2021-03-09T08:57:52.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0457f0 2021-03-09T08:57:52.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.930+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:52.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:52.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:52.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:57:57.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc098910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:57.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc098910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:57.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc098910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc064f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:57.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc098910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc064f00).stop 2021-03-09T08:57:57.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc098910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:57.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc098910 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:57.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:57.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:57.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:57.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc005e30 2021-03-09T08:57:57.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c1f0 con 0x7f21ec09c280 2021-03-09T08:57:57.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05b910 con 0x7f21ec0a7f50 2021-03-09T08:57:57.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:57:57.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:57.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:57:57.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:57.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:57.934 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1103 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:57:57.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:57.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:57.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:57.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:57.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01df50 con 0x7f21ec0a7f50 2021-03-09T08:57:57.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0637d0 con 0x7f21ec0a7f50 2021-03-09T08:57:57.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f21e0009030 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:57.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e005dea0 con 0x7f21ec0a7f50 2021-03-09T08:57:57.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a96e0 con 0x7f21ec0a7f50 2021-03-09T08:57:57.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T08:57:57.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:57:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:57:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:57:57.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:02.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:02.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f21e0009030 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f21e0009030 tx=0x7f21e0003820).stop 2021-03-09T08:58:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:02.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc096740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035080 con 0x7f21cc005e30 2021-03-09T08:58:02.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec09c280 2021-03-09T08:58:02.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fd10 con 0x7f21cc0457f0 2021-03-09T08:58:02.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:02.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:02.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:02.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:02.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc096740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:02.931 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1140 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:02.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:02.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:02.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:02.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:02.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08f660 con 0x7f21cc0457f0 2021-03-09T08:58:02.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01df50 con 0x7f21cc0457f0 2021-03-09T08:58:02.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc096740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f21e00a5eb0 tx=0x7f21e0079b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:02.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0457f0 2021-03-09T08:58:02.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0457f0 2021-03-09T08:58:02.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0037080 con 0x7f21cc0457f0 2021-03-09T08:58:02.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:02.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:02.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:02.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc096740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:58:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc096740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc096740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f21e00a5eb0 tx=0x7f21e0079b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:07.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc096740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f21e00a5eb0 tx=0x7f21e0079b40).stop 2021-03-09T08:58:07.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc096740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:07.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc096740 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:07.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:07.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:07.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:07.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05b910 con 0x7f21cc005e30 2021-03-09T08:58:07.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec09c280 2021-03-09T08:58:07.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025800 con 0x7f21ec0a7f50 2021-03-09T08:58:07.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:07.930 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1176 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:07.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:07.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:07.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:07.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:07.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:07.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:07.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:07.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:07.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03db50 con 0x7f21ec0a7f50 2021-03-09T08:58:07.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T08:58:07.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f21e0056a60 tx=0x7f21e0056b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:07.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0056070 con 0x7f21ec0a7f50 2021-03-09T08:58:07.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21ec0a7f50 2021-03-09T08:58:07.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0037070 con 0x7f21ec0a7f50 2021-03-09T08:58:07.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:07.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:07.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:07.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:12.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:12.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:12.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f21e0056a60 tx=0x7f21e0056b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:12.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f21e0056a60 tx=0x7f21e0056b40).stop 2021-03-09T08:58:12.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc039580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:12.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc039580 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:12.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:12.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc013bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:12.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:12.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc005e30 2021-03-09T08:58:12.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02aad0 con 0x7f21ec09c280 2021-03-09T08:58:12.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc005850 con 0x7f21cc0457f0 2021-03-09T08:58:12.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:12.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:12.931 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1211 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:12.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:12.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc013bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:12.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:12.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:12.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:12.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:12.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:12.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc045110 con 0x7f21cc0457f0 2021-03-09T08:58:12.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0457f0 2021-03-09T08:58:12.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc013bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f21e409a460 tx=0x7f21e4059f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:12.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21cc0457f0 2021-03-09T08:58:12.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4056480 con 0x7f21cc0457f0 2021-03-09T08:58:12.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40317b0 con 0x7f21cc0457f0 2021-03-09T08:58:12.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:12.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:12.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:12.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:16.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:58:16.836 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:58:17.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc013bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:58:17.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc013bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:17.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc013bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f21e409a460 tx=0x7f21e4059f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:17.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc013bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f21e409a460 tx=0x7f21e4059f40).stop 2021-03-09T08:58:17.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc013bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:17.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc013bf0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:17.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:17.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0396c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:17.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:17.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040310 con 0x7f21cc005e30 2021-03-09T08:58:17.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025800 con 0x7f21ec09c280 2021-03-09T08:58:17.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:17.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec0a7f50 2021-03-09T08:58:17.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:17.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:17.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:17.933 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1248 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:17.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0396c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:17.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:17.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:17.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:17.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:17.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:58:17.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc045110 con 0x7f21ec0a7f50 2021-03-09T08:58:17.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0396c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f21e4031fa0 tx=0x7f21e4024d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:17.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21ec0a7f50 2021-03-09T08:58:17.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e405b3f0 con 0x7f21ec0a7f50 2021-03-09T08:58:17.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4056480 con 0x7f21ec0a7f50 2021-03-09T08:58:17.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.938+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:17.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:17.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:17.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:22.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0396c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:22.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0396c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:22.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0396c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f21e4031fa0 tx=0x7f21e4024d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:22.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0396c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f21e4031fa0 tx=0x7f21e4024d90).stop 2021-03-09T08:58:22.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0396c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:22.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0396c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:22.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc02edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f280 con 0x7f21cc005e30 2021-03-09T08:58:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc005850 con 0x7f21ec09c280 2021-03-09T08:58:22.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02dc40 con 0x7f21cc0457f0 2021-03-09T08:58:22.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:22.933 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1284 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:22.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:22.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:22.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:22.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc02edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:22.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:22.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:22.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:22.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:22.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc010740 con 0x7f21cc0457f0 2021-03-09T08:58:22.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0457f0 2021-03-09T08:58:22.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc02edc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f21cc045710 tx=0x7f21e4097c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:22.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21cc0457f0 2021-03-09T08:58:22.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e405d4f0 con 0x7f21cc0457f0 2021-03-09T08:58:22.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e405d0b0 con 0x7f21cc0457f0 2021-03-09T08:58:22.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.938+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:22.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:22.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:22.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:27.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc02edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:27.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc02edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:27.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc02edc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f21cc045710 tx=0x7f21e4097c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:27.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc02edc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f21cc045710 tx=0x7f21e4097c20).stop 2021-03-09T08:58:27.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 msgr2=0x7f21cc02edc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:27.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0457f0 0x7f21cc02edc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:27.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc038500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21cc005e30 2021-03-09T08:58:27.933 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1320 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:27.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec09c280 2021-03-09T08:58:27.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f260 con 0x7f21ec0a7f50 2021-03-09T08:58:27.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:27.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:27.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:27.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:27.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc038500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:27.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:27.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:27.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:27.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:27.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:58:27.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc010740 con 0x7f21ec0a7f50 2021-03-09T08:58:27.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc038500 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f21e4031560 tx=0x7f21e409d450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:27.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009b50 con 0x7f21ec0a7f50 2021-03-09T08:58:27.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40240d0 con 0x7f21ec0a7f50 2021-03-09T08:58:27.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40970a0 con 0x7f21ec0a7f50 2021-03-09T08:58:27.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.938+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:27.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:27.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:27.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:32.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc038500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:32.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc038500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:32.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc038500 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f21e4031560 tx=0x7f21e409d450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:32.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc038500 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f21e4031560 tx=0x7f21e409d450).stop 2021-03-09T08:58:32.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc038500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:32.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc038500 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:32.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:32.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:32.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc028a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:32.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a40 con 0x7f21cc005e30 2021-03-09T08:58:32.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec09c280 2021-03-09T08:58:32.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042e90 con 0x7f21cc0285d0 2021-03-09T08:58:32.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:32.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:32.935 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1356 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:32.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:32.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:32.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc028a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:32.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:32.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:32.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:32.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:32.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02fcc0 con 0x7f21cc0285d0 2021-03-09T08:58:32.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc098000 con 0x7f21cc0285d0 2021-03-09T08:58:32.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc028a50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f21dc064c90 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:32.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc064af0 con 0x7f21cc0285d0 2021-03-09T08:58:32.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc064af0 con 0x7f21cc0285d0 2021-03-09T08:58:32.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T08:58:32.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.938+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:32.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:32.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:32.938+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:37.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc028a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:58:37.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc028a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:37.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc028a50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f21dc064c90 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:37.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc028a50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f21dc064c90 tx=0x7f21dc02dfc0).stop 2021-03-09T08:58:37.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc028a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:37.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc028a50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:37.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:37.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:37.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:37.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21cc005e30 2021-03-09T08:58:37.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc077300 con 0x7f21ec09c280 2021-03-09T08:58:37.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a900 con 0x7f21ec0a7f50 2021-03-09T08:58:37.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:37.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:37.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:37.939 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1393 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:37.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:37.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:37.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:37.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:37.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:37.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:37.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dc80 con 0x7f21ec0a7f50 2021-03-09T08:58:37.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02fcc0 con 0x7f21ec0a7f50 2021-03-09T08:58:37.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d2f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f21dc05b630 tx=0x7f21dc009580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:37.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21ec0a7f50 2021-03-09T08:58:37.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T08:58:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0200e0 con 0x7f21ec0a7f50 2021-03-09T08:58:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.938+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.942+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:37.942+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:42.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:42.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:42.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d2f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f21dc05b630 tx=0x7f21dc009580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:42.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d2f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f21dc05b630 tx=0x7f21dc009580).stop 2021-03-09T08:58:42.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03d2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:42.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d2f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:42.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc005e30 2021-03-09T08:58:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a40 con 0x7f21ec09c280 2021-03-09T08:58:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042e90 con 0x7f21cc0285d0 2021-03-09T08:58:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:42.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:42.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:42.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:42.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:42.939 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:42.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1431 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:42.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:42.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:42.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:42.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:42.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc034fb0 con 0x7f21cc0285d0 2021-03-09T08:58:42.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01dc80 con 0x7f21cc0285d0 2021-03-09T08:58:42.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098dd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f21dc06b0b0 tx=0x7f21dc06b200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:42.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21cc0285d0 2021-03-09T08:58:42.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc036970 con 0x7f21cc0285d0 2021-03-09T08:58:42.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0860b0 con 0x7f21cc0285d0 2021-03-09T08:58:42.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.942+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:42.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:42.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:42.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:46.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:58:47.135 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:58:47.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc098dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:58:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc098dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098dd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f21dc06b0b0 tx=0x7f21dc06b200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098dd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f21dc06b0b0 tx=0x7f21dc06b200).stop 2021-03-09T08:58:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc098dd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098dd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:47.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21cc005e30 2021-03-09T08:58:47.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013680 con 0x7f21ec09c280 2021-03-09T08:58:47.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e60 con 0x7f21ec0a7f50 2021-03-09T08:58:47.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:47.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:47.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:47.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:47.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:47.939 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1464 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:47.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:47.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:47.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:47.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:47.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc004c60 con 0x7f21ec0a7f50 2021-03-09T08:58:47.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03db50 con 0x7f21ec0a7f50 2021-03-09T08:58:47.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.938+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03be10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f21e0079e20 tx=0x7f21e0079cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:47.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a5fb0 con 0x7f21ec0a7f50 2021-03-09T08:58:47.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006b90 con 0x7f21ec0a7f50 2021-03-09T08:58:47.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T08:58:47.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.942+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:47.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.942+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:47.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:47.942+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:52.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:52.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03be10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f21e0079e20 tx=0x7f21e0079cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03be10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f21e0079e20 tx=0x7f21e0079cb0).stop 2021-03-09T08:58:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03be10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:52.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03be10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a40 con 0x7f21cc005e30 2021-03-09T08:58:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042e90 con 0x7f21ec09c280 2021-03-09T08:58:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc0285d0 2021-03-09T08:58:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:52.940 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1501 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dc80 con 0x7f21cc0285d0 2021-03-09T08:58:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21cc0285d0 2021-03-09T08:58:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.938+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098a90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f21e0077a50 tx=0x7f21e00a6020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0077660 con 0x7f21cc0285d0 2021-03-09T08:58:52.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0077480 con 0x7f21cc0285d0 2021-03-09T08:58:52.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007ced0 con 0x7f21cc0285d0 2021-03-09T08:58:52.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.946+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:52.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:58:57.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc098a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:57.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc098a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:57.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098a90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f21e0077a50 tx=0x7f21e00a6020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098a90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f21e0077a50 tx=0x7f21e00a6020).stop 2021-03-09T08:58:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc098a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc098a90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:57.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:57.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:57.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21cc005e30 2021-03-09T08:58:57.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a780 con 0x7f21ec09c280 2021-03-09T08:58:57.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037e50 con 0x7f21ec0a7f50 2021-03-09T08:58:57.940 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1542 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:58:57.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:58:57.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:57.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:58:57.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:57.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:57.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:57.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:57.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:57.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:57.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03cd90 con 0x7f21ec0a7f50 2021-03-09T08:58:57.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:58:57.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c8a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f21e4059f10 tx=0x7f21e4009b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:57.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40318e0 con 0x7f21ec0a7f50 2021-03-09T08:58:57.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40543f0 con 0x7f21ec0a7f50 2021-03-09T08:58:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cd10 con 0x7f21ec0a7f50 2021-03-09T08:58:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.946+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:58:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:58:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:58:57.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:02.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:02.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:02.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c8a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f21e4059f10 tx=0x7f21e4009b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:02.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c8a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f21e4059f10 tx=0x7f21e4009b50).stop 2021-03-09T08:59:02.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:02.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c8a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:02.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:02.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:02.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:02.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089e80 con 0x7f21cc005e30 2021-03-09T08:59:02.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02aad0 con 0x7f21ec09c280 2021-03-09T08:59:02.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc0285d0 2021-03-09T08:59:02.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:02.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:02.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:02.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:02.944 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1576 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:02.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:02.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:02.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:02.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:02.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:02.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T08:59:02.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc034fb0 con 0x7f21cc0285d0 2021-03-09T08:59:02.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025210 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f21dc05b400 tx=0x7f21dc05da20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:02.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21cc0285d0 2021-03-09T08:59:02.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc036970 con 0x7f21cc0285d0 2021-03-09T08:59:02.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0285d0 2021-03-09T08:59:02.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.946+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:02.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:02.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:02.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:07.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc025210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:59:07.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc025210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:07.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025210 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f21dc05b400 tx=0x7f21dc05da20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:07.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025210 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f21dc05b400 tx=0x7f21dc05da20).stop 2021-03-09T08:59:07.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc025210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:07.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025210 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:07.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:07.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:07.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:07.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03de10 con 0x7f21cc005e30 2021-03-09T08:59:07.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21ec09c280 2021-03-09T08:59:07.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012310 con 0x7f21ec0a7f50 2021-03-09T08:59:07.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:07.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:07.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:07.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:07.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:07.944 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1610 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:07.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:07.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:07.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:07.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:07.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012450 con 0x7f21ec0a7f50 2021-03-09T08:59:07.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03cd90 con 0x7f21ec0a7f50 2021-03-09T08:59:07.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f21e4059f80 tx=0x7f21e409d390).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:07.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T08:59:07.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009390 con 0x7f21ec0a7f50 2021-03-09T08:59:07.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T08:59:07.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.950+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:07.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:07.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:07.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:12.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:12.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:12.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f21e4059f80 tx=0x7f21e409d390).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:12.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f21e4059f80 tx=0x7f21e409d390).stop 2021-03-09T08:59:12.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:12.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021070 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:12.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc097db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:12.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:12.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:12.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0912d0 con 0x7f21cc005e30 2021-03-09T08:59:12.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec09c280 2021-03-09T08:59:12.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0126a0 con 0x7f21cc0285d0 2021-03-09T08:59:12.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:12.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:12.944 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1644 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:12.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:12.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:12.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.938+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc097db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:12.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:12.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:12.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:12.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:12.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039f10 con 0x7f21cc0285d0 2021-03-09T08:59:12.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01dc80 con 0x7f21cc0285d0 2021-03-09T08:59:12.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc097db0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f21e0077820 tx=0x7f21e007c3d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:12.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0285d0 2021-03-09T08:59:12.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006b90 con 0x7f21cc0285d0 2021-03-09T08:59:12.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007ced0 con 0x7f21cc0285d0 2021-03-09T08:59:12.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.954+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:12.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:17.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:59:17.186 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:59:17.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc097db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:17.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc097db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:17.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc097db0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f21e0077820 tx=0x7f21e007c3d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:17.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc097db0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f21e0077820 tx=0x7f21e007c3d0).stop 2021-03-09T08:59:17.942 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1683 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:17.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc097db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:17.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc097db0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:17.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:17.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:17.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc044110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:17.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21cc005e30 2021-03-09T08:59:17.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046690 con 0x7f21ec09c280 2021-03-09T08:59:17.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc020a00 con 0x7f21ec0a7f50 2021-03-09T08:59:17.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:17.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:17.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc044110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:17.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:17.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:17.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:17.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:17.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:17.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:17.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044600 con 0x7f21ec0a7f50 2021-03-09T08:59:17.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:59:17.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc044110 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f21dc086820 tx=0x7f21dc065e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:17.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc037cd0 con 0x7f21ec0a7f50 2021-03-09T08:59:17.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T08:59:17.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc084130 con 0x7f21ec0a7f50 2021-03-09T08:59:17.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.950+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:17.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:17.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:17.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:22.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc044110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:59:22.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc044110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:22.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc044110 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f21dc086820 tx=0x7f21dc065e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:22.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc044110 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f21dc086820 tx=0x7f21dc065e90).stop 2021-03-09T08:59:22.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc044110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:22.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc044110 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:22.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035280 con 0x7f21cc005e30 2021-03-09T08:59:22.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033ec0 con 0x7f21ec09c280 2021-03-09T08:59:22.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039f60 con 0x7f21cc0285d0 2021-03-09T08:59:22.945 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1718 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:22.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:22.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:22.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:22.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:22.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:22.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:22.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:22.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:22.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:22.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T08:59:22.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0285d0 2021-03-09T08:59:22.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04b580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f21e00a8f50 tx=0x7f21e000b4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:22.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0026640 con 0x7f21cc0285d0 2021-03-09T08:59:22.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0026640 con 0x7f21cc0285d0 2021-03-09T08:59:22.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0285d0 2021-03-09T08:59:22.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.950+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:22.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:22.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:22.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:27.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:59:27.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:27.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04b580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f21e00a8f50 tx=0x7f21e000b4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:27.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04b580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f21e00a8f50 tx=0x7f21e000b4d0).stop 2021-03-09T08:59:27.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04b580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:27.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04b580 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:27.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:27.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:27.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:27.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0126a0 con 0x7f21cc005e30 2021-03-09T08:59:27.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec09c280 2021-03-09T08:59:27.946 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1753 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:27.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21ec0a7f50 2021-03-09T08:59:27.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:27.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:27.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:27.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:27.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:27.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:27.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:27.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:27.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:27.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08e330 con 0x7f21ec0a7f50 2021-03-09T08:59:27.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012450 con 0x7f21ec0a7f50 2021-03-09T08:59:27.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f21e4051760 tx=0x7f21e4035410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:27.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T08:59:27.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T08:59:27.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4050f30 con 0x7f21ec0a7f50 2021-03-09T08:59:27.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.950+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:27.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:27.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:27.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:32.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:32.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:32.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f21e4051760 tx=0x7f21e4035410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:32.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f21e4051760 tx=0x7f21e4035410).stop 2021-03-09T08:59:32.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:32.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:32.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:32.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:32.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:32.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02cf50 con 0x7f21cc005e30 2021-03-09T08:59:32.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:32.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f50 con 0x7f21ec09c280 2021-03-09T08:59:32.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:32.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0444c0 con 0x7f21cc0285d0 2021-03-09T08:59:32.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:32.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:32.948 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1788 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:32.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:32.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:32.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:32.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:32.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:32.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030a90 con 0x7f21cc0285d0 2021-03-09T08:59:32.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044600 con 0x7f21cc0285d0 2021-03-09T08:59:32.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04a170 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0346c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:32.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0285d0 2021-03-09T08:59:32.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21cc0285d0 2021-03-09T08:59:32.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0285d0 2021-03-09T08:59:32.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.954+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:32.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:32.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:32.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:37.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:59:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04a170 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0346c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04a170 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0346c0).stop 2021-03-09T08:59:37.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04a170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:37.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04a170 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:37.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:37.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:37.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:37.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0272b0 con 0x7f21cc005e30 2021-03-09T08:59:37.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039f60 con 0x7f21ec09c280 2021-03-09T08:59:37.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec0a7f50 2021-03-09T08:59:37.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:37.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:37.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:37.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:37.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:37.949 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1826 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:37.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:37.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:37.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:37.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:37.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056d60 con 0x7f21ec0a7f50 2021-03-09T08:59:37.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030a90 con 0x7f21ec0a7f50 2021-03-09T08:59:37.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc028d70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:37.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T08:59:37.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T08:59:37.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0280d0 con 0x7f21ec0a7f50 2021-03-09T08:59:37.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:37.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:37.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:37.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:42.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:59:42.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:42.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc028d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:42.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc028d70).stop 2021-03-09T08:59:42.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:42.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099dc0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:42.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:42.948 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1860 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:42.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:42.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:42.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041f10 con 0x7f21cc005e30 2021-03-09T08:59:42.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046690 con 0x7f21ec09c280 2021-03-09T08:59:42.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cbf0 con 0x7f21cc0285d0 2021-03-09T08:59:42.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:42.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:42.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:42.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:42.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:42.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:42.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:42.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:42.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:42.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03dbc0 con 0x7f21cc0285d0 2021-03-09T08:59:42.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T08:59:42.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010de0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f21e0009100 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:42.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0285d0 2021-03-09T08:59:42.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006b90 con 0x7f21cc0285d0 2021-03-09T08:59:42.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0008e20 con 0x7f21cc0285d0 2021-03-09T08:59:42.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.954+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:42.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:42.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:42.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:47.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:59:47.228 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T08:59:47.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc010de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:47.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc010de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:47.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010de0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f21e0009100 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:47.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010de0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f21e0009100 tx=0x7f21e007c5a0).stop 2021-03-09T08:59:47.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc010de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:47.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010de0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:47.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:47.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:47.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:47.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc020a00 con 0x7f21cc005e30 2021-03-09T08:59:47.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21ec09c280 2021-03-09T08:59:47.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec0a7f50 2021-03-09T08:59:47.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:47.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:47.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:47.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:47.951 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1900 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:47.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:47.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:47.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:47.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:47.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:47.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:59:47.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056d60 con 0x7f21ec0a7f50 2021-03-09T08:59:47.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f21cc02fe70 tx=0x7f21dc028910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:47.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21ec0a7f50 2021-03-09T08:59:47.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0349b0 con 0x7f21ec0a7f50 2021-03-09T08:59:47.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T08:59:47.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:47.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:47.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:47.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:52.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f21cc02fe70 tx=0x7f21dc028910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f21cc02fe70 tx=0x7f21dc028910).stop 2021-03-09T08:59:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:52.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc004530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:52.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:52.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:52.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc061170 con 0x7f21cc005e30 2021-03-09T08:59:52.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046690 con 0x7f21ec09c280 2021-03-09T08:59:52.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0475e0 con 0x7f21cc0285d0 2021-03-09T08:59:52.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:52.951 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1932 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:52.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:52.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:52.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:52.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc004530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:52.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:52.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:52.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:52.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:52.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0103e0 con 0x7f21cc0285d0 2021-03-09T08:59:52.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03dbc0 con 0x7f21cc0285d0 2021-03-09T08:59:52.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc004530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f21e001ead0 tx=0x7f21e001ea40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:52.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0285d0 2021-03-09T08:59:52.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0285d0 2021-03-09T08:59:52.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00999a0 con 0x7f21cc0285d0 2021-03-09T08:59:52.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:52.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:52.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:52.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T08:59:57.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc004530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:57.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc004530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:57.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc004530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f21e001ead0 tx=0x7f21e001ea40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:57.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc004530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f21e001ead0 tx=0x7f21e001ea40).stop 2021-03-09T08:59:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc004530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc004530 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:57.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:57.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:57.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:57.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097640 con 0x7f21cc005e30 2021-03-09T08:59:57.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024bc0 con 0x7f21ec09c280 2021-03-09T08:59:57.956 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1969 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T08:59:57.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010c80 con 0x7f21ec0a7f50 2021-03-09T08:59:57.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T08:59:57.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T08:59:57.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:57.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:57.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:57.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:57.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:57.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:57.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:57.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc081470 con 0x7f21ec0a7f50 2021-03-09T08:59:57.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T08:59:57.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc061420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:57.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T08:59:57.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f040 con 0x7f21ec0a7f50 2021-03-09T08:59:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T08:59:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T08:59:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T08:59:57.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T08:59:57.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:02.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:02.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:02.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc061420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:02.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc061420).stop 2021-03-09T09:00:02.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:02.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:02.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:02.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:02.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:02.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046890 con 0x7f21cc005e30 2021-03-09T09:00:02.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05d320 con 0x7f21ec09c280 2021-03-09T09:00:02.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e710 con 0x7f21cc0285d0 2021-03-09T09:00:02.954 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2005 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:02.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:02.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:02.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:02.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:02.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:02.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:02.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:02.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:02.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:02.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c6e0 con 0x7f21cc0285d0 2021-03-09T09:00:02.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08e330 con 0x7f21cc0285d0 2021-03-09T09:00:02.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04af20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f21cc035540 tx=0x7f21e4031130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:02.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21cc0285d0 2021-03-09T09:00:02.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409d120 con 0x7f21cc0285d0 2021-03-09T09:00:02.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400c5f0 con 0x7f21cc0285d0 2021-03-09T09:00:02.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:02.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:02.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:02.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:07.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:00:07.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:07.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04af20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f21cc035540 tx=0x7f21e4031130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:07.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04af20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f21cc035540 tx=0x7f21e4031130).stop 2021-03-09T09:00:07.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04af20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:07.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04af20 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:07.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:07.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:07.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:07.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00efe0 con 0x7f21cc005e30 2021-03-09T09:00:07.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec09c280 2021-03-09T09:00:07.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e710 con 0x7f21ec0a7f50 2021-03-09T09:00:07.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:07.955 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2040 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:07.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:07.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:07.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:07.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:07.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:07.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:07.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:07.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:07.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f840 con 0x7f21ec0a7f50 2021-03-09T09:00:07.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c6e0 con 0x7f21ec0a7f50 2021-03-09T09:00:07.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f21cc016f40 tx=0x7f21e4009bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:07.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409d0e0 con 0x7f21ec0a7f50 2021-03-09T09:00:07.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409d0e0 con 0x7f21ec0a7f50 2021-03-09T09:00:07.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402ba90 con 0x7f21ec0a7f50 2021-03-09T09:00:07.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.962+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:07.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.962+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:07.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:07.962+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:12.951 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2077 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:12.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:00:12.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f21cc016f40 tx=0x7f21e4009bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f21cc016f40 tx=0x7f21e4009bc0).stop 2021-03-09T09:00:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21cc005e30 2021-03-09T09:00:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089980 con 0x7f21ec09c280 2021-03-09T09:00:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0555c0 con 0x7f21cc0285d0 2021-03-09T09:00:12.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:12.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:12.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:12.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:12.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:12.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:12.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:00:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc081470 con 0x7f21cc0285d0 2021-03-09T09:00:12.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09a290 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0178a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:12.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21cc0285d0 2021-03-09T09:00:12.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0090a0 con 0x7f21cc0285d0 2021-03-09T09:00:12.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc064190 con 0x7f21cc0285d0 2021-03-09T09:00:12.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:12.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:12.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:12.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:17.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:00:17.271 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:00:17.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:17.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:17.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09a290 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0178a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:17.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09a290 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0178a0).stop 2021-03-09T09:00:17.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09a290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:17.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09a290 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:17.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:17.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:17.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:17.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038050 con 0x7f21cc005e30 2021-03-09T09:00:17.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:17.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:17.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:17.956 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2111 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:17.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:17.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0309a0 con 0x7f21ec09c280 2021-03-09T09:00:17.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046690 con 0x7f21ec0a7f50 2021-03-09T09:00:17.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:17.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:17.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:17.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:17.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:17.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00a920 con 0x7f21ec0a7f50 2021-03-09T09:00:17.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:00:17.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc071f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:17.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017030 con 0x7f21ec0a7f50 2021-03-09T09:00:17.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21ec0a7f50 2021-03-09T09:00:17.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017030 con 0x7f21ec0a7f50 2021-03-09T09:00:17.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:17.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:17.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:17.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:22.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:22.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:22.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc071f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:22.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc071f10).stop 2021-03-09T09:00:22.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:22.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:22.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:22.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:22.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:22.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022370 con 0x7f21cc005e30 2021-03-09T09:00:22.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fd30 con 0x7f21ec09c280 2021-03-09T09:00:22.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049af0 con 0x7f21cc0285d0 2021-03-09T09:00:22.957 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2148 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:22.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:22.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:22.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:22.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:22.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:22.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:22.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:22.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:22.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:22.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095590 con 0x7f21cc0285d0 2021-03-09T09:00:22.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f840 con 0x7f21cc0285d0 2021-03-09T09:00:22.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025de0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f21e4095440 tx=0x7f21e4009b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:22.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21cc0285d0 2021-03-09T09:00:22.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3530 con 0x7f21cc0285d0 2021-03-09T09:00:22.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402ba90 con 0x7f21cc0285d0 2021-03-09T09:00:22.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:22.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:22.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:22.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:27.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc025de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:27.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc025de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:27.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025de0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f21e4095440 tx=0x7f21e4009b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:27.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025de0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f21e4095440 tx=0x7f21e4009b20).stop 2021-03-09T09:00:27.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc025de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:27.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc025de0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:27.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:27.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:27.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:27.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025770 con 0x7f21cc005e30 2021-03-09T09:00:27.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e40 con 0x7f21ec09c280 2021-03-09T09:00:27.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092530 con 0x7f21ec0a7f50 2021-03-09T09:00:27.957 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2183 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:27.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:27.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:27.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:27.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:27.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:27.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:27.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:27.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:27.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:27.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092810 con 0x7f21ec0a7f50 2021-03-09T09:00:27.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095590 con 0x7f21ec0a7f50 2021-03-09T09:00:27.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f21e409f040 tx=0x7f21e409eae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:27.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21ec0a7f50 2021-03-09T09:00:27.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4052900 con 0x7f21ec0a7f50 2021-03-09T09:00:27.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e050 con 0x7f21ec0a7f50 2021-03-09T09:00:27.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:27.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:27.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:27.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:32.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:32.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:32.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f21e409f040 tx=0x7f21e409eae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:32.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f21e409f040 tx=0x7f21e409eae0).stop 2021-03-09T09:00:32.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:32.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:32.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:32.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:32.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:32.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f870 con 0x7f21cc005e30 2021-03-09T09:00:32.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f50 con 0x7f21ec09c280 2021-03-09T09:00:32.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0926d0 con 0x7f21cc0285d0 2021-03-09T09:00:32.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:32.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:32.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:32.959 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2221 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:32.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:32.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:32.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:32.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:32.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:32.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:32.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:00:32.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0103e0 con 0x7f21cc0285d0 2021-03-09T09:00:32.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.958+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010a30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:32.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:00:32.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:00:32.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21cc0285d0 2021-03-09T09:00:32.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:32.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:32.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:32.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:37.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc010a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:37.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc010a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:37.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010a30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:37.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010a30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b3e0).stop 2021-03-09T09:00:37.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc010a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:37.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc010a30 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:37.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:37.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:37.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:37.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0926d0 con 0x7f21cc005e30 2021-03-09T09:00:37.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044ec0 con 0x7f21ec09c280 2021-03-09T09:00:37.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056c50 con 0x7f21ec0a7f50 2021-03-09T09:00:37.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:37.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:37.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:37.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:37.961 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2255 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:37.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:37.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:37.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:37.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:37.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:37.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016db0 con 0x7f21ec0a7f50 2021-03-09T09:00:37.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00a920 con 0x7f21ec0a7f50 2021-03-09T09:00:37.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f21cc03c3f0 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:37.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0691f0 con 0x7f21ec0a7f50 2021-03-09T09:00:37.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0691f0 con 0x7f21ec0a7f50 2021-03-09T09:00:37.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21ec0a7f50 2021-03-09T09:00:37.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:37.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.970+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:37.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:37.970+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:42.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:00:42.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:42.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f21cc03c3f0 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:42.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f21cc03c3f0 tx=0x7f21dc02df50).stop 2021-03-09T09:00:42.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:42.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:42.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:42.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:42.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0890b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:42.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f080 con 0x7f21cc005e30 2021-03-09T09:00:42.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b6c0 con 0x7f21ec09c280 2021-03-09T09:00:42.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0108f0 con 0x7f21cc0285d0 2021-03-09T09:00:42.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:42.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:42.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:42.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:42.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0890b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:42.964 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2292 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:42.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:42.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:42.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:42.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:42.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08ab50 con 0x7f21cc0285d0 2021-03-09T09:00:42.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016db0 con 0x7f21cc0285d0 2021-03-09T09:00:42.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.961+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0890b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc05a800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:42.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065d80 con 0x7f21cc0285d0 2021-03-09T09:00:42.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05a0e0 con 0x7f21cc0285d0 2021-03-09T09:00:42.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc04d640 con 0x7f21cc0285d0 2021-03-09T09:00:42.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.965+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:42.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:42.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:42.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:47.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:00:47.315 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:00:47.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0890b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:00:47.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0890b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:47.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0890b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc05a800).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:47.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0890b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc05a800).stop 2021-03-09T09:00:47.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0890b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:47.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0890b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:47.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:47.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:47.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:47.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:47.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:47.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ae30 con 0x7f21cc005e30 2021-03-09T09:00:47.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025770 con 0x7f21ec09c280 2021-03-09T09:00:47.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016c30 con 0x7f21ec0a7f50 2021-03-09T09:00:47.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:47.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:47.964 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2327 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:47.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:47.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:47.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:47.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:47.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:47.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ee30 con 0x7f21ec0a7f50 2021-03-09T09:00:47.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08ab50 con 0x7f21ec0a7f50 2021-03-09T09:00:47.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f21dc050ca0 tx=0x7f21dc0509a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:47.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05aa40 con 0x7f21ec0a7f50 2021-03-09T09:00:47.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:00:47.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:00:47.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.968+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:47.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:47.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:47.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:52.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:52.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:52.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f21dc050ca0 tx=0x7f21dc0509a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:52.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f21dc050ca0 tx=0x7f21dc0509a0).stop 2021-03-09T09:00:52.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:52.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:52.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:52.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:52.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:52.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044ec0 con 0x7f21cc005e30 2021-03-09T09:00:52.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019e00 con 0x7f21ec09c280 2021-03-09T09:00:52.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025770 con 0x7f21cc0285d0 2021-03-09T09:00:52.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:52.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:52.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:52.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:52.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:52.968 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2364 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:52.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:52.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:52.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:52.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:52.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc019fd0 con 0x7f21cc0285d0 2021-03-09T09:00:52.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ee30 con 0x7f21cc0285d0 2021-03-09T09:00:52.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f21dc04f970 tx=0x7f21dc010690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:52.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc04fee0 con 0x7f21cc0285d0 2021-03-09T09:00:52.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc04fee0 con 0x7f21cc0285d0 2021-03-09T09:00:52.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc04fee0 con 0x7f21cc0285d0 2021-03-09T09:00:52.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.968+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:52.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:52.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:52.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:00:57.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:00:57.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:57.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f21dc04f970 tx=0x7f21dc010690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:57.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f470 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f21dc04f970 tx=0x7f21dc010690).stop 2021-03-09T09:00:57.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00f470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:57.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f470 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:57.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:57.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:57.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc018e30 con 0x7f21cc005e30 2021-03-09T09:00:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc003de0 con 0x7f21ec09c280 2021-03-09T09:00:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043f90 con 0x7f21ec0a7f50 2021-03-09T09:00:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:00:57.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:00:57.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:57.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:57.969 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2399 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:00:57.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:57.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:57.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:57.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:57.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:57.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00a920 con 0x7f21ec0a7f50 2021-03-09T09:00:57.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc092810 con 0x7f21ec0a7f50 2021-03-09T09:00:57.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f21cc00d420 tx=0x7f21e4009890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:57.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089090 con 0x7f21ec0a7f50 2021-03-09T09:00:57.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40529a0 con 0x7f21ec0a7f50 2021-03-09T09:00:57.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e070 con 0x7f21ec0a7f50 2021-03-09T09:00:57.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.975+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:00:57.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:00:57.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:00:57.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:02.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:02.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:02.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f21cc00d420 tx=0x7f21e4009890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:02.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f21cc00d420 tx=0x7f21e4009890).stop 2021-03-09T09:01:02.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:02.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:02.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:02.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc011330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:02.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:02.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036fc0 con 0x7f21cc005e30 2021-03-09T09:01:02.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094d10 con 0x7f21ec09c280 2021-03-09T09:01:02.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019c80 con 0x7f21cc0285d0 2021-03-09T09:01:02.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:02.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:02.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:02.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:02.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc011330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:02.971 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:02.963+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2436 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:02.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:02.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:02.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:02.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:02.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0578f0 con 0x7f21cc0285d0 2021-03-09T09:01:02.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00a920 con 0x7f21cc0285d0 2021-03-09T09:01:02.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc011330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f21cc03b820 tx=0x7f21e4095230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:02.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.971+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089080 con 0x7f21cc0285d0 2021-03-09T09:01:02.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.971+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400c310 con 0x7f21cc0285d0 2021-03-09T09:01:02.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400c480 con 0x7f21cc0285d0 2021-03-09T09:01:02.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.975+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:02.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:02.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:02.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:07.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc011330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:07.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc011330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc011330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f21cc03b820 tx=0x7f21e4095230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc011330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f21cc03b820 tx=0x7f21e4095230).stop 2021-03-09T09:01:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc011330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc011330 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017010 con 0x7f21cc005e30 2021-03-09T09:01:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c450 con 0x7f21ec09c280 2021-03-09T09:01:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033900 con 0x7f21ec0a7f50 2021-03-09T09:01:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:07.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:07.971 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2474 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:07.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:07.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:07.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:07.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:07.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:07.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0897f0 con 0x7f21ec0a7f50 2021-03-09T09:01:07.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc019fd0 con 0x7f21ec0a7f50 2021-03-09T09:01:07.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f21dc006d40 tx=0x7f21dc0530c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:07.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21ec0a7f50 2021-03-09T09:01:07.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21ec0a7f50 2021-03-09T09:01:07.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.974+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:01:07.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.974+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:07.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.978+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:07.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:07.978+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:12.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:01:12.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:12.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f21dc006d40 tx=0x7f21dc0530c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:12.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f21dc006d40 tx=0x7f21dc0530c0).stop 2021-03-09T09:01:12.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:12.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:12.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0175d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:12.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:12.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:12.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21cc005e30 2021-03-09T09:01:12.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026d90 con 0x7f21ec09c280 2021-03-09T09:01:12.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c450 con 0x7f21cc0285d0 2021-03-09T09:01:12.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:12.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:12.972 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2509 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:12.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0175d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:12.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:12.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc008370 con 0x7f21cc0285d0 2021-03-09T09:01:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.966+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:01:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0175d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f21e00381e0 tx=0x7f21e007d5c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:01:12.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0037df0 con 0x7f21cc0285d0 2021-03-09T09:01:12.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.974+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0037090 con 0x7f21cc0285d0 2021-03-09T09:01:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.974+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.974+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:12.974+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:17.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:01:17.357 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:01:17.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0175d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:17.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0175d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:17.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0175d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f21e00381e0 tx=0x7f21e007d5c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:17.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0175d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f21e00381e0 tx=0x7f21e007d5c0).stop 2021-03-09T09:01:17.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0175d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:17.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0175d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:17.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:17.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:17.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:17.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21cc005e30 2021-03-09T09:01:17.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04db90 con 0x7f21ec09c280 2021-03-09T09:01:17.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026d90 con 0x7f21ec0a7f50 2021-03-09T09:01:17.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:17.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:17.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:17.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:17.975 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2547 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:17.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:17.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:17.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:17.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:17.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:17.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:01:17.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0578f0 con 0x7f21ec0a7f50 2021-03-09T09:01:17.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f21e405bba0 tx=0x7f21e405bb10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:17.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401ec00 con 0x7f21ec0a7f50 2021-03-09T09:01:17.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401e830 con 0x7f21ec0a7f50 2021-03-09T09:01:17.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e070 con 0x7f21ec0a7f50 2021-03-09T09:01:17.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.977+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:17.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.977+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:17.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:17.977+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:22.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:01:22.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:22.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f21e405bba0 tx=0x7f21e405bb10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:22.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f21e405bba0 tx=0x7f21e405bb10).stop 2021-03-09T09:01:22.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:22.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:22.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:22.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:22.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:22.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0939d0 con 0x7f21cc005e30 2021-03-09T09:01:22.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc003de0 con 0x7f21ec09c280 2021-03-09T09:01:22.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049c50 con 0x7f21cc0285d0 2021-03-09T09:01:22.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:22.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:22.976 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2579 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:22.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:22.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:22.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:22.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:22.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:22.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:22.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:22.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0402a0 con 0x7f21cc0285d0 2021-03-09T09:01:22.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0897f0 con 0x7f21cc0285d0 2021-03-09T09:01:22.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f21dc055470 tx=0x7f21dc009180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:22.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:01:22.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:01:22.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:01:22.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.981+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:22.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:22.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:22.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:27.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:27.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:27.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f21dc055470 tx=0x7f21dc009180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:27.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f21dc055470 tx=0x7f21dc009180).stop 2021-03-09T09:01:27.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:27.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:27.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:27.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:27.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:27.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c450 con 0x7f21cc005e30 2021-03-09T09:01:27.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21ec09c280 2021-03-09T09:01:27.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b9f0 con 0x7f21ec0a7f50 2021-03-09T09:01:27.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:27.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:27.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:27.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:27.977 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2615 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:27.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:27.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:27.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:27.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:27.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:27.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0103e0 con 0x7f21ec0a7f50 2021-03-09T09:01:27.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:01:27.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21e4020060 tx=0x7f21e40354f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:27.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:01:27.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e405bbd0 con 0x7f21ec0a7f50 2021-03-09T09:01:27.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e405bbd0 con 0x7f21ec0a7f50 2021-03-09T09:01:27.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.977+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:27.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:27.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:27.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:32.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:32.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:32.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21e4020060 tx=0x7f21e40354f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:32.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21e4020060 tx=0x7f21e40354f0).stop 2021-03-09T09:01:32.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:32.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:32.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:32.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:32.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:32.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017d90 con 0x7f21cc005e30 2021-03-09T09:01:32.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21ec09c280 2021-03-09T09:01:32.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053360 con 0x7f21cc0285d0 2021-03-09T09:01:32.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:32.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:32.978 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2651 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:32.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:32.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:32.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.969+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:32.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:32.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:32.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:32.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:32.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:01:32.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc008370 con 0x7f21cc0285d0 2021-03-09T09:01:32.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f21e001e8a0 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:32.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:01:32.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:01:32.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21cc0285d0 2021-03-09T09:01:32.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.981+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:32.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:32.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:32.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:37.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:37.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:37.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f21e001e8a0 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:37.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f21e001e8a0 tx=0x7f21e0003820).stop 2021-03-09T09:01:37.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:37.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:37.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:37.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:37.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:37.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048a30 con 0x7f21cc005e30 2021-03-09T09:01:37.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b9f0 con 0x7f21ec09c280 2021-03-09T09:01:37.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d080 con 0x7f21ec0a7f50 2021-03-09T09:01:37.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:37.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:37.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:37.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:37.980 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2687 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:37.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:37.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:37.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:37.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:37.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:37.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04bb50 con 0x7f21ec0a7f50 2021-03-09T09:01:37.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0402a0 con 0x7f21ec0a7f50 2021-03-09T09:01:37.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f21dc06b0b0 tx=0x7f21dc05a1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:37.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:01:37.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21ec0a7f50 2021-03-09T09:01:37.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:01:37.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.980+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:37.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:37.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:37.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:42.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:42.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:42.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f21dc06b0b0 tx=0x7f21dc05a1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:42.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f21dc06b0b0 tx=0x7f21dc05a1f0).stop 2021-03-09T09:01:42.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:42.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:42.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:42.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc008c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:42.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:42.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21cc005e30 2021-03-09T09:01:42.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c60 con 0x7f21ec09c280 2021-03-09T09:01:42.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d950 con 0x7f21cc0285d0 2021-03-09T09:01:42.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:42.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:42.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:42.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:42.981 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2725 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:42.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc008c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:42.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:42.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:42.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:42.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:42.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02b960 con 0x7f21cc0285d0 2021-03-09T09:01:42.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0103e0 con 0x7f21cc0285d0 2021-03-09T09:01:42.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc008c40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f21e40532e0 tx=0x7f21e4085790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:42.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4085ea0 con 0x7f21cc0285d0 2021-03-09T09:01:42.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40097b0 con 0x7f21cc0285d0 2021-03-09T09:01:42.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e080 con 0x7f21cc0285d0 2021-03-09T09:01:42.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.980+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:42.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.980+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:42.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:42.980+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:47.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:01:47.402 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:01:47.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:47.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:47.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc008c40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f21e40532e0 tx=0x7f21e4085790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:47.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc008c40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f21e40532e0 tx=0x7f21e4085790).stop 2021-03-09T09:01:47.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc008c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:47.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc008c40 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:47.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:47.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:47.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:47.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21cc005e30 2021-03-09T09:01:47.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089980 con 0x7f21ec09c280 2021-03-09T09:01:47.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b560 con 0x7f21ec0a7f50 2021-03-09T09:01:47.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:47.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:47.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:47.988 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2762 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:47.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:47.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:47.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:47.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:47.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:47.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040d30 con 0x7f21ec0a7f50 2021-03-09T09:01:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04bb50 con 0x7f21ec0a7f50 2021-03-09T09:01:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f21dc0283e0 tx=0x7f21dc028290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:47.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:01:47.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:01:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21ec0a7f50 2021-03-09T09:01:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.980+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:47.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:47.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:47.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:52.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:01:52.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:52.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f21dc0283e0 tx=0x7f21dc028290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:52.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f21dc0283e0 tx=0x7f21dc028290).stop 2021-03-09T09:01:52.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:52.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:52.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:52.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:52.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:52.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056830 con 0x7f21cc005e30 2021-03-09T09:01:52.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025770 con 0x7f21ec09c280 2021-03-09T09:01:52.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21cc0285d0 2021-03-09T09:01:52.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:52.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:52.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:52.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:52.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:52.989 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:52.972+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2794 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:52.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:52.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:52.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:52.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:52.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0899d0 con 0x7f21cc0285d0 2021-03-09T09:01:52.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02b960 con 0x7f21cc0285d0 2021-03-09T09:01:52.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00ee50 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f21e4096ff0 tx=0x7f21e4085c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:52.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4085ea0 con 0x7f21cc0285d0 2021-03-09T09:01:52.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e405fa00 con 0x7f21cc0285d0 2021-03-09T09:01:52.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e070 con 0x7f21cc0285d0 2021-03-09T09:01:52.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.980+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:52.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:52.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:52.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:01:57.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:57.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:57.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00ee50 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f21e4096ff0 tx=0x7f21e4085c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:57.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00ee50 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f21e4096ff0 tx=0x7f21e4085c30).stop 2021-03-09T09:01:57.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00ee50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:57.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00ee50 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:57.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:57.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:57.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:57.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02c290 con 0x7f21cc005e30 2021-03-09T09:01:57.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010650 con 0x7f21ec09c280 2021-03-09T09:01:57.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a3f0 con 0x7f21ec0a7f50 2021-03-09T09:01:57.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:01:57.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:57.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:01:57.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:57.986 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2831 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:01:57.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:57.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:57.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:57.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:57.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:57.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043f40 con 0x7f21ec0a7f50 2021-03-09T09:01:57.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0899d0 con 0x7f21ec0a7f50 2021-03-09T09:01:57.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f21cc04ea90 tx=0x7f21e4085b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:57.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T09:01:57.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009ac0 con 0x7f21ec0a7f50 2021-03-09T09:01:57.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21ec0a7f50 2021-03-09T09:01:57.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.984+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:01:57.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:01:57.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:01:57.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:02.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:02.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:02.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f21cc04ea90 tx=0x7f21e4085b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:02.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f21cc04ea90 tx=0x7f21e4085b50).stop 2021-03-09T09:02:02.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc099da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:02.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc099da0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:02.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:02.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:02.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:02.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0450c0 con 0x7f21cc005e30 2021-03-09T09:02:02.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025770 con 0x7f21ec09c280 2021-03-09T09:02:02.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053c00 con 0x7f21cc0285d0 2021-03-09T09:02:02.987 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2865 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:02.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:02.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:02.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:02.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:02.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:02.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:02.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:02.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:02.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:02.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc005800 con 0x7f21cc0285d0 2021-03-09T09:02:02.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043f40 con 0x7f21cc0285d0 2021-03-09T09:02:02.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f410 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f21cc04e8c0 tx=0x7f21e4096b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:02.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089090 con 0x7f21cc0285d0 2021-03-09T09:02:02.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4080d40 con 0x7f21cc0285d0 2021-03-09T09:02:02.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409b000 con 0x7f21cc0285d0 2021-03-09T09:02:02.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.983+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:02.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.987+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:02.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:02.987+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:07.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:07.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:07.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f410 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f21cc04e8c0 tx=0x7f21e4096b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:07.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f410 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f21cc04e8c0 tx=0x7f21e4096b70).stop 2021-03-09T09:02:07.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00f410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:07.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00f410 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:07.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:07.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:07.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:07.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057f70 con 0x7f21cc005e30 2021-03-09T09:02:07.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056830 con 0x7f21ec09c280 2021-03-09T09:02:07.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a3f0 con 0x7f21ec0a7f50 2021-03-09T09:02:07.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:07.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:07.987 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2903 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:07.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:07.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:07.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.979+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:07.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:07.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:07.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:07.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:07.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08ee00 con 0x7f21ec0a7f50 2021-03-09T09:02:07.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:02:07.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043030 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e007d670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:07.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:02:07.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21ec0a7f50 2021-03-09T09:02:07.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21ec0a7f50 2021-03-09T09:02:07.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.991+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:07.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:07.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:07.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:12.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:12.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:12.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043030 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e007d670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:12.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043030 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e007d670).stop 2021-03-09T09:02:12.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:12.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043030 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:12.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:12.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:12.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:12.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a890 con 0x7f21cc005e30 2021-03-09T09:02:12.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0133a0 con 0x7f21ec09c280 2021-03-09T09:02:12.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004370 con 0x7f21cc0285d0 2021-03-09T09:02:12.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:12.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:12.993 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2940 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:12.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:12.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:12.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:12.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:12.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:12.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:12.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:13.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:02:13.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040d30 con 0x7f21cc0285d0 2021-03-09T09:02:13.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043520 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f21cc04ab50 tx=0x7f21dc05a130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:13.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc0285d0 2021-03-09T09:02:13.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a10 con 0x7f21cc0285d0 2021-03-09T09:02:13.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc0285d0 2021-03-09T09:02:13.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.991+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:13.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:13.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:12.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:17.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:02:17.439 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:02:17.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc043520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:17.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc043520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:17.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043520 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f21cc04ab50 tx=0x7f21dc05a130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:17.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043520 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f21cc04ab50 tx=0x7f21dc05a130).stop 2021-03-09T09:02:17.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc043520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:17.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043520 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:17.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:17.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:17.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:17.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057f70 con 0x7f21cc005e30 2021-03-09T09:02:17.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004370 con 0x7f21ec09c280 2021-03-09T09:02:17.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e700 con 0x7f21ec0a7f50 2021-03-09T09:02:17.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:17.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:17.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:17.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:17.991 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2978 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:17.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:17.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:17.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:17.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:17.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:17.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc008190 con 0x7f21ec0a7f50 2021-03-09T09:02:17.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc005800 con 0x7f21ec0a7f50 2021-03-09T09:02:17.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03e200 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f21e40275f0 tx=0x7f21e4089d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:17.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:02:17.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409be10 con 0x7f21ec0a7f50 2021-03-09T09:02:17.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e040 con 0x7f21ec0a7f50 2021-03-09T09:02:17.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.991+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:17.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:17.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:17.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:22.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:22.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:22.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03e200 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f21e40275f0 tx=0x7f21e4089d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:22.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03e200 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f21e40275f0 tx=0x7f21e4089d60).stop 2021-03-09T09:02:22.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03e200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:22.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03e200 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:22.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc035960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a3d0 con 0x7f21cc005e30 2021-03-09T09:02:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057f70 con 0x7f21ec09c280 2021-03-09T09:02:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056830 con 0x7f21cc0285d0 2021-03-09T09:02:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc035960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:22.992 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3012 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0222a0 con 0x7f21cc0285d0 2021-03-09T09:02:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08ee00 con 0x7f21cc0285d0 2021-03-09T09:02:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.987+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc035960 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00373b0 con 0x7f21cc0285d0 2021-03-09T09:02:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0285d0 2021-03-09T09:02:22.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21cc0285d0 2021-03-09T09:02:22.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.991+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:22.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.995+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:22.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:22.995+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:27.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc035960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:27.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc035960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:27.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc035960 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:27.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc035960 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d540).stop 2021-03-09T09:02:27.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc035960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:27.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc035960 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:27.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:27.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:27.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:27.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057f70 con 0x7f21cc005e30 2021-03-09T09:02:27.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056830 con 0x7f21ec09c280 2021-03-09T09:02:27.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a3d0 con 0x7f21ec0a7f50 2021-03-09T09:02:27.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:27.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:27.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:27.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:27.993 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:27.983+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3047 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:27.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:27.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:27.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:27.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:27.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:27.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc005800 con 0x7f21ec0a7f50 2021-03-09T09:02:27.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc008190 con 0x7f21ec0a7f50 2021-03-09T09:02:27.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035e50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f21cc0208b0 tx=0x7f21e40039a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:27.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089030 con 0x7f21ec0a7f50 2021-03-09T09:02:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4085630 con 0x7f21ec0a7f50 2021-03-09T09:02:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4033af0 con 0x7f21ec0a7f50 2021-03-09T09:02:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.999+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:28.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:28.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:27.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:32.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc035e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:02:32.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc035e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:32.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035e50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f21cc0208b0 tx=0x7f21e40039a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:32.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035e50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f21cc0208b0 tx=0x7f21e40039a0).stop 2021-03-09T09:02:32.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc035e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:32.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035e50 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:32.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:32.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc036340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:32.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:32.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d4c0 con 0x7f21cc005e30 2021-03-09T09:02:32.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c3f0 con 0x7f21ec09c280 2021-03-09T09:02:32.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094170 con 0x7f21cc0285d0 2021-03-09T09:02:32.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:32.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:32.994 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3083 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:32.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:32.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:32.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc036340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:32.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:32.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:32.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:32.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:32.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f300 con 0x7f21cc0285d0 2021-03-09T09:02:32.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc005800 con 0x7f21cc0285d0 2021-03-09T09:02:32.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc036340 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f21e4033580 tx=0x7f21e4053d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:32.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089030 con 0x7f21cc0285d0 2021-03-09T09:02:32.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4061400 con 0x7f21cc0285d0 2021-03-09T09:02:33.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e080 con 0x7f21cc0285d0 2021-03-09T09:02:33.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.999+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:33.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:33.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:32.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:37.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc036340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:37.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc036340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:37.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc036340 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f21e4033580 tx=0x7f21e4053d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:37.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc036340 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f21e4033580 tx=0x7f21e4053d30).stop 2021-03-09T09:02:37.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc036340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:37.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc036340 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:37.994 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3121 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:37.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:37.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:37.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:37.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049c50 con 0x7f21cc005e30 2021-03-09T09:02:37.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:37.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21ec09c280 2021-03-09T09:02:37.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049ea0 con 0x7f21ec0a7f50 2021-03-09T09:02:37.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:37.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:37.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:37.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:37.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:37.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:37.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:38.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:38.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04dad0 con 0x7f21ec0a7f50 2021-03-09T09:02:38.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f300 con 0x7f21ec0a7f50 2021-03-09T09:02:38.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030b70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f21cc04add0 tx=0x7f21e40804a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:38.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:02:38.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4089a90 con 0x7f21ec0a7f50 2021-03-09T09:02:38.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4064140 con 0x7f21ec0a7f50 2021-03-09T09:02:38.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.998+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:38.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:38.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:37.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:42.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:42.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:42.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030b70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f21cc04add0 tx=0x7f21e40804a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:42.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030b70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f21cc04add0 tx=0x7f21e40804a0).stop 2021-03-09T09:02:42.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:42.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030b70 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:42.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0092c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:42.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:42.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:42.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088c40 con 0x7f21cc005e30 2021-03-09T09:02:42.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c650 con 0x7f21ec09c280 2021-03-09T09:02:42.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036ee0 con 0x7f21cc0285d0 2021-03-09T09:02:42.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:42.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:42.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:42.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:42.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:42.997 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3153 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:42.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:42.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:42.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:42.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:42.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093700 con 0x7f21cc0285d0 2021-03-09T09:02:42.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0222a0 con 0x7f21cc0285d0 2021-03-09T09:02:43.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0092c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e00a6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:43.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:02:43.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:42.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fd20 con 0x7f21cc0285d0 2021-03-09T09:02:43.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:43.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21cc0285d0 2021-03-09T09:02:43.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:43.002+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:43.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:43.002+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:43.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:43.002+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:47.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:02:47.481 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:02:47.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:47.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:47.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0092c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e00a6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0092c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e00a6200).stop 2021-03-09T09:02:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0092c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0092c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:47.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc009f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:47.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:47.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d990 con 0x7f21cc005e30 2021-03-09T09:02:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f90 con 0x7f21ec09c280 2021-03-09T09:02:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011a60 con 0x7f21ec0a7f50 2021-03-09T09:02:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:47.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:47.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:47.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:47.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc009f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:47.999 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3191 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:48.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:48.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:48.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:48.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:48.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:02:48.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093700 con 0x7f21ec0a7f50 2021-03-09T09:02:48.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.994+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc009f50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f21e00550b0 tx=0x7f21e007c5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:48.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ccc0 con 0x7f21ec0a7f50 2021-03-09T09:02:48.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0054e80 con 0x7f21ec0a7f50 2021-03-09T09:02:48.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007ccc0 con 0x7f21ec0a7f50 2021-03-09T09:02:48.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.998+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:48.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:48.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:47.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:52.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc009f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:52.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc009f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:52.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc009f50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f21e00550b0 tx=0x7f21e007c5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:52.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc009f50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f21e00550b0 tx=0x7f21e007c5b0).stop 2021-03-09T09:02:52.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc009f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:52.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc009f50 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:52.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:52.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:52.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:52.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044b30 con 0x7f21cc005e30 2021-03-09T09:02:53.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048850 con 0x7f21ec09c280 2021-03-09T09:02:53.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046150 con 0x7f21cc0285d0 2021-03-09T09:02:53.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:53.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:53.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:53.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:53.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:53.002 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:52.990+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3231 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:53.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:53.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:53.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:53.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:53.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0274a0 con 0x7f21cc0285d0 2021-03-09T09:02:53.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04dad0 con 0x7f21cc0285d0 2021-03-09T09:02:53.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00b210 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f21e405d1c0 tx=0x7f21e409e210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:53.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21cc0285d0 2021-03-09T09:02:53.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4080210 con 0x7f21cc0285d0 2021-03-09T09:02:53.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4014610 con 0x7f21cc0285d0 2021-03-09T09:02:53.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.998+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:53.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:53.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:52.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:02:57.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:57.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:57.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00b210 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f21e405d1c0 tx=0x7f21e409e210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:57.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00b210 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f21e405d1c0 tx=0x7f21e409e210).stop 2021-03-09T09:02:57.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc00b210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:57.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc00b210 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:57.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:57.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:57.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:58.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042d50 con 0x7f21cc005e30 2021-03-09T09:02:58.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0349e0 con 0x7f21ec09c280 2021-03-09T09:02:58.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21ec0a7f50 2021-03-09T09:02:58.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:02:58.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:58.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:02:58.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:58.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:58.002 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3264 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:02:58.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:58.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:58.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:58.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:58.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053160 con 0x7f21ec0a7f50 2021-03-09T09:02:58.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0274a0 con 0x7f21ec0a7f50 2021-03-09T09:02:58.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.998+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c650 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f21e4087da0 tx=0x7f21e4087c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:58.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21ec0a7f50 2021-03-09T09:02:58.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:57.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4087e10 con 0x7f21ec0a7f50 2021-03-09T09:02:58.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:58.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4080070 con 0x7f21ec0a7f50 2021-03-09T09:02:58.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:58.002+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:02:58.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:58.002+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:02:58.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:02:58.002+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:02.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:02.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:03.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c650 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f21e4087da0 tx=0x7f21e4087c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:03.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c650 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f21e4087da0 tx=0x7f21e4087c50).stop 2021-03-09T09:03:03.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:03.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c650 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:03.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:03.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:03.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc013b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:03.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02b260 con 0x7f21cc005e30 2021-03-09T09:03:03.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03aef0 con 0x7f21ec09c280 2021-03-09T09:03:03.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c740 con 0x7f21cc0285d0 2021-03-09T09:03:03.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:03.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:03.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:03.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:03.006 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3297 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:03.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc013b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:03.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:03.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:03.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:03.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:03.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047980 con 0x7f21cc0285d0 2021-03-09T09:03:03.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:03:03.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc013b00 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:03.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:02.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01eea0 con 0x7f21cc0285d0 2021-03-09T09:03:03.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:03.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01ec70 con 0x7f21cc0285d0 2021-03-09T09:03:03.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:03.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc084200 con 0x7f21cc0285d0 2021-03-09T09:03:03.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:03.006+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:03.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:03.006+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:03.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:03.006+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:08.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc013b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:03:08.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc013b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:08.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc013b00 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:08.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc013b00 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).stop 2021-03-09T09:03:08.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc013b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:08.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc013b00 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:08.002 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3337 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:08.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:08.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:08.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:08.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ac10 con 0x7f21cc005e30 2021-03-09T09:03:08.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0920f0 con 0x7f21ec09c280 2021-03-09T09:03:08.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048890 con 0x7f21ec0a7f50 2021-03-09T09:03:08.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:08.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:08.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:08.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:08.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:08.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:08.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:08.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:08.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:08.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025720 con 0x7f21ec0a7f50 2021-03-09T09:03:08.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:07.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc047980 con 0x7f21ec0a7f50 2021-03-09T09:03:08.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013ff0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc086900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:08.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06acd0 con 0x7f21ec0a7f50 2021-03-09T09:03:08.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21ec0a7f50 2021-03-09T09:03:08.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc086ac0 con 0x7f21ec0a7f50 2021-03-09T09:03:08.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:08.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:08.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:08.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:13.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:13.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:13.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013ff0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc086900).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:13.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013ff0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc086900).stop 2021-03-09T09:03:13.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:13.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013ff0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:13.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:13.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:13.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc014e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:13.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:13.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21cc005e30 2021-03-09T09:03:13.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054300 con 0x7f21ec09c280 2021-03-09T09:03:13.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:13.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0255a0 con 0x7f21cc0285d0 2021-03-09T09:03:13.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:13.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:13.006 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3369 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:13.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc014e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:13.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:13.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:13.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:13.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:13.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:03:13.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:12.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025720 con 0x7f21cc0285d0 2021-03-09T09:03:13.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc014e60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f21dc07fe40 tx=0x7f21dc04ec00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:13.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc093b60 con 0x7f21cc0285d0 2021-03-09T09:03:13.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc093b60 con 0x7f21cc0285d0 2021-03-09T09:03:13.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:03:13.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:13.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:13.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:13.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:17.521 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:03:17.525 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:03:18.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc014e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:18.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc014e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:18.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc014e60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f21dc07fe40 tx=0x7f21dc04ec00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:18.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc014e60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f21dc07fe40 tx=0x7f21dc04ec00).stop 2021-03-09T09:03:18.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc014e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:18.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc014e60 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:18.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:18.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:18.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:18.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022400 con 0x7f21cc005e30 2021-03-09T09:03:18.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0229c0 con 0x7f21ec09c280 2021-03-09T09:03:18.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0173f0 con 0x7f21ec0a7f50 2021-03-09T09:03:18.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:18.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:18.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:18.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:18.006 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3411 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:18.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:17.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:18.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:18.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:18.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:18.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:18.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043f40 con 0x7f21ec0a7f50 2021-03-09T09:03:18.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:03:18.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e7c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f21dc06b420 tx=0x7f21dc01eb70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:18.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc093b60 con 0x7f21ec0a7f50 2021-03-09T09:03:18.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc093b60 con 0x7f21ec0a7f50 2021-03-09T09:03:18.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:03:18.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:18.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:18.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:18.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:23.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:23.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:23.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e7c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f21dc06b420 tx=0x7f21dc01eb70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:23.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e7c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f21dc06b420 tx=0x7f21dc01eb70).stop 2021-03-09T09:03:23.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00e7c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:23.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e7c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:23.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0158f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:23.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:23.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:23.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056830 con 0x7f21cc005e30 2021-03-09T09:03:23.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0920f0 con 0x7f21ec09c280 2021-03-09T09:03:23.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21cc0285d0 2021-03-09T09:03:23.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:23.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:23.008 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3443 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:23.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:23.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:23.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:22.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0158f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:23.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:23.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:23.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:23.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:23.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021af0 con 0x7f21cc0285d0 2021-03-09T09:03:23.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21cc0285d0 2021-03-09T09:03:23.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0158f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e0006bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:23.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000b3e0 con 0x7f21cc0285d0 2021-03-09T09:03:23.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a5e10 con 0x7f21cc0285d0 2021-03-09T09:03:23.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21cc0285d0 2021-03-09T09:03:23.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:23.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:23.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:23.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:28.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0158f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:28.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0158f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:28.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0158f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e0006bd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:28.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0158f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e0006bd0).stop 2021-03-09T09:03:28.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0158f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:28.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0158f0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:28.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:28.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:27.998+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:28.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:28.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094db0 con 0x7f21cc005e30 2021-03-09T09:03:28.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055fb0 con 0x7f21ec09c280 2021-03-09T09:03:28.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037eb0 con 0x7f21ec0a7f50 2021-03-09T09:03:28.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:28.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:28.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:28.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:28.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:28.009 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3483 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:28.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:28.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:28.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:28.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:28.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03aef0 con 0x7f21ec0a7f50 2021-03-09T09:03:28.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021af0 con 0x7f21ec0a7f50 2021-03-09T09:03:28.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d050 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e007cae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:28.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0026470 con 0x7f21ec0a7f50 2021-03-09T09:03:28.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00265d0 con 0x7f21ec0a7f50 2021-03-09T09:03:28.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0026470 con 0x7f21ec0a7f50 2021-03-09T09:03:28.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:28.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:28.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:28.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:33.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:33.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:33.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d050 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e007cae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:33.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d050 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e007cae0).stop 2021-03-09T09:03:33.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03d050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03d050 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049ff0 con 0x7f21cc005e30 2021-03-09T09:03:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0411e0 con 0x7f21ec09c280 2021-03-09T09:03:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0173f0 con 0x7f21cc0285d0 2021-03-09T09:03:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:33.011 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04ac10 con 0x7f21cc0285d0 2021-03-09T09:03:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03aef0 con 0x7f21cc0285d0 2021-03-09T09:03:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.006+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09b860 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f21e009d850 tx=0x7f21e005da30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003cba0 con 0x7f21cc0285d0 2021-03-09T09:03:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e005d0d0 con 0x7f21cc0285d0 2021-03-09T09:03:33.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0037090 con 0x7f21cc0285d0 2021-03-09T09:03:33.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:33.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.014+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:33.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:33.014+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:38.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:38.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:38.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09b860 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f21e009d850 tx=0x7f21e005da30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:38.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09b860 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f21e009d850 tx=0x7f21e005da30).stop 2021-03-09T09:03:38.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09b860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:38.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09b860 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:38.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:38.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:38.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:38.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21cc005e30 2021-03-09T09:03:38.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ac10 con 0x7f21ec09c280 2021-03-09T09:03:38.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d0b0 con 0x7f21ec0a7f50 2021-03-09T09:03:38.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:38.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:38.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:38.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:38.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:38.012 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3551 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:38.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:38.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:38.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:38.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:38.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02b040 con 0x7f21ec0a7f50 2021-03-09T09:03:38.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053160 con 0x7f21ec0a7f50 2021-03-09T09:03:38.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09bd50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e4053f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:38.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4014b60 con 0x7f21ec0a7f50 2021-03-09T09:03:38.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4014520 con 0x7f21ec0a7f50 2021-03-09T09:03:38.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4014690 con 0x7f21ec0a7f50 2021-03-09T09:03:38.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.009+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:38.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.013+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:38.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:38.013+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:43.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:43.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:43.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09bd50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e4053f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:43.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09bd50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e4053f40).stop 2021-03-09T09:03:43.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:43.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09bd50 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:43.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:43.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:43.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:43.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0430c0 con 0x7f21cc005e30 2021-03-09T09:03:43.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0173f0 con 0x7f21ec09c280 2021-03-09T09:03:43.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f5a0 con 0x7f21cc0285d0 2021-03-09T09:03:43.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:43.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:43.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:43.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:43.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:43.014 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3585 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:43.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:43.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:43.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:43.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:43.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01bf90 con 0x7f21cc0285d0 2021-03-09T09:03:43.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04ac10 con 0x7f21cc0285d0 2021-03-09T09:03:43.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0539a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007cae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:43.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0097e70 con 0x7f21cc0285d0 2021-03-09T09:03:43.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0097e70 con 0x7f21cc0285d0 2021-03-09T09:03:43.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0058020 con 0x7f21cc0285d0 2021-03-09T09:03:43.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.017+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:43.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.017+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:43.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:43.017+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:47.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:03:47.597 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:03:48.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:48.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:48.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0539a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007cae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:48.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0539a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007cae0).stop 2021-03-09T09:03:48.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0539a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:48.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0539a0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:48.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:48.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:48.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:48.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21cc005e30 2021-03-09T09:03:48.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053860 con 0x7f21ec09c280 2021-03-09T09:03:48.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039af0 con 0x7f21ec0a7f50 2021-03-09T09:03:48.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:48.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:48.015 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3622 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:48.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:48.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:48.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.005+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:48.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:48.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:48.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:48.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:48.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc051860 con 0x7f21ec0a7f50 2021-03-09T09:03:48.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043f40 con 0x7f21ec0a7f50 2021-03-09T09:03:48.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097f50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f21dc07fe40 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:48.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc067000 con 0x7f21ec0a7f50 2021-03-09T09:03:48.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc016bc0 con 0x7f21ec0a7f50 2021-03-09T09:03:48.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:03:48.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.013+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:48.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.017+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:48.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:48.017+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:53.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc097f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:03:53.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc097f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:53.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097f50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f21dc07fe40 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:53.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097f50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f21dc07fe40 tx=0x7f21dc02df50).stop 2021-03-09T09:03:53.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc097f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:53.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc097f50 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:53.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:53.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:53.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:53.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d4c0 con 0x7f21cc005e30 2021-03-09T09:03:53.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0173f0 con 0x7f21ec09c280 2021-03-09T09:03:53.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010160 con 0x7f21cc0285d0 2021-03-09T09:03:53.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:53.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:53.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:53.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:53.016 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3659 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:53.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc024c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:53.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:53.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:53.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:53.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:53.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:03:53.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc051860 con 0x7f21cc0285d0 2021-03-09T09:03:53.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc024c30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:53.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f0e0 con 0x7f21cc0285d0 2021-03-09T09:03:53.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f0e0 con 0x7f21cc0285d0 2021-03-09T09:03:53.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21cc0285d0 2021-03-09T09:03:53.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.017+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:53.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:53.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:53.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:03:58.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc024c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:58.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc024c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:58.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc024c30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:58.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc024c30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc080000).stop 2021-03-09T09:03:58.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc024c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:58.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc024c30 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:58.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:58.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:58.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:58.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21cc005e30 2021-03-09T09:03:58.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00efd0 con 0x7f21ec09c280 2021-03-09T09:03:58.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039af0 con 0x7f21ec0a7f50 2021-03-09T09:03:58.017 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3696 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:03:58.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:03:58.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:58.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:03:58.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:58.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:58.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:58.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:58.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:58.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:58.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0920d0 con 0x7f21ec0a7f50 2021-03-09T09:03:58.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02b040 con 0x7f21ec0a7f50 2021-03-09T09:03:58.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.013+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0531b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f21cc0031a0 tx=0x7f21e4080050).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:58.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408e480 con 0x7f21ec0a7f50 2021-03-09T09:03:58.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408e600 con 0x7f21ec0a7f50 2021-03-09T09:03:58.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e070 con 0x7f21ec0a7f50 2021-03-09T09:03:58.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.017+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:03:58.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:03:58.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:03:58.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:03.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:03.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:03.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0531b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f21cc0031a0 tx=0x7f21e4080050).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:03.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0531b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f21cc0031a0 tx=0x7f21e4080050).stop 2021-03-09T09:04:03.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0531b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:03.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0531b0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:03.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:03.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:03.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:03.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028190 con 0x7f21cc005e30 2021-03-09T09:04:03.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d270 con 0x7f21ec09c280 2021-03-09T09:04:03.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21cc0285d0 2021-03-09T09:04:03.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:03.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:03.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:03.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:03.018 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:03.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:03.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:03.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:03.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:03.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:03.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089b80 con 0x7f21cc0285d0 2021-03-09T09:04:03.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:04:03.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc027700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:03.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a10 con 0x7f21cc0285d0 2021-03-09T09:04:03.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a10 con 0x7f21cc0285d0 2021-03-09T09:04:03.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:04:03.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.021+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:03.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:03.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:03.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:08.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:08.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:08.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc027700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:08.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc027700).stop 2021-03-09T09:04:08.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc099f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:08.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:08.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:08.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:08.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:08.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055fb0 con 0x7f21cc005e30 2021-03-09T09:04:08.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a4b0 con 0x7f21ec09c280 2021-03-09T09:04:08.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21ec0a7f50 2021-03-09T09:04:08.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:08.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:08.020 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3770 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:08.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:08.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:08.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:08.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:08.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:08.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:08.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:08.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0426f0 con 0x7f21ec0a7f50 2021-03-09T09:04:08.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089b80 con 0x7f21ec0a7f50 2021-03-09T09:04:08.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043620 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc094a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:08.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc055070 con 0x7f21ec0a7f50 2021-03-09T09:04:08.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0551d0 con 0x7f21ec0a7f50 2021-03-09T09:04:08.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:04:08.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.021+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:08.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.025+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:08.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:08.025+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:13.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:13.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:13.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043620 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc094a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:13.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043620 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc094a30).stop 2021-03-09T09:04:13.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:13.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043620 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:13.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:13.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:13.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:13.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ff70 con 0x7f21cc005e30 2021-03-09T09:04:13.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039af0 con 0x7f21ec09c280 2021-03-09T09:04:13.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028190 con 0x7f21cc0285d0 2021-03-09T09:04:13.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:13.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:13.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:13.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:13.021 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3801 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:13.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:13.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:13.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:13.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:13.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:13.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041190 con 0x7f21cc0285d0 2021-03-09T09:04:13.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0426f0 con 0x7f21cc0285d0 2021-03-09T09:04:13.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc009150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:13.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc04fc00 con 0x7f21cc0285d0 2021-03-09T09:04:13.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc027aa0 con 0x7f21cc0285d0 2021-03-09T09:04:13.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc027aa0 con 0x7f21cc0285d0 2021-03-09T09:04:13.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.021+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:13.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:13.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:13.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:17.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:04:17.654 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:04:18.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:18.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:18.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc009150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:18.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc009150).stop 2021-03-09T09:04:18.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc099f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:18.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc099f60 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:18.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:18.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:18.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc036390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:18.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f1b0 con 0x7f21cc005e30 2021-03-09T09:04:18.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0413e0 con 0x7f21ec09c280 2021-03-09T09:04:18.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ddd0 con 0x7f21ec0a7f50 2021-03-09T09:04:18.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:18.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:18.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:18.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:18.029 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3840 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:18.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc036390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:18.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:18.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:18.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:18.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:18.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01be00 con 0x7f21ec0a7f50 2021-03-09T09:04:18.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.013+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041190 con 0x7f21ec0a7f50 2021-03-09T09:04:18.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc036390 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f21dc06a7f0 tx=0x7f21dc060510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:18.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc04fc00 con 0x7f21ec0a7f50 2021-03-09T09:04:18.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc097070 con 0x7f21ec0a7f50 2021-03-09T09:04:18.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:04:18.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.021+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:18.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:18.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:18.021+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:23.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc036390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:04:23.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc036390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:23.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc036390 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f21dc06a7f0 tx=0x7f21dc060510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:23.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc036390 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f21dc06a7f0 tx=0x7f21dc060510).stop 2021-03-09T09:04:23.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc036390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:23.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc036390 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:23.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:23.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:23.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:23.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024250 con 0x7f21cc005e30 2021-03-09T09:04:23.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21ec09c280 2021-03-09T09:04:23.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057250 con 0x7f21cc0285d0 2021-03-09T09:04:23.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:23.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:23.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:23.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:23.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:23.024 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3873 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:23.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:23.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:23.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:23.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:23.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:04:23.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01be00 con 0x7f21cc0285d0 2021-03-09T09:04:23.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0987a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f21dc0277b0 tx=0x7f21dc06ec00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:23.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc076700 con 0x7f21cc0285d0 2021-03-09T09:04:23.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06e960 con 0x7f21cc0285d0 2021-03-09T09:04:23.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0285d0 2021-03-09T09:04:23.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.025+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:23.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.025+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:23.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:23.025+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:28.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:04:28.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:28.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0987a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f21dc0277b0 tx=0x7f21dc06ec00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:28.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0987a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f21dc0277b0 tx=0x7f21dc06ec00).stop 2021-03-09T09:04:28.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0987a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:28.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0987a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:28.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:28.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:28.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:28.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03aa80 con 0x7f21cc005e30 2021-03-09T09:04:28.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03e8e0 con 0x7f21ec09c280 2021-03-09T09:04:28.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0413e0 con 0x7f21ec0a7f50 2021-03-09T09:04:28.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:28.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:28.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:28.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:28.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:28.026 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3915 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:28.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:28.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:28.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:28.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:28.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc019420 con 0x7f21ec0a7f50 2021-03-09T09:04:28.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01bf90 con 0x7f21ec0a7f50 2021-03-09T09:04:28.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.021+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09aca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e000b470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:28.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0037de0 con 0x7f21ec0a7f50 2021-03-09T09:04:28.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:04:28.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:04:28.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.029+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:28.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.029+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:28.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:28.029+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:33.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:33.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:33.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09aca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e000b470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:33.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09aca0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e000b470).stop 2021-03-09T09:04:33.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09aca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:33.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09aca0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:33.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:33.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:33.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:33.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fdf0 con 0x7f21cc005e30 2021-03-09T09:04:33.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21ec09c280 2021-03-09T09:04:33.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0333e0 con 0x7f21cc0285d0 2021-03-09T09:04:33.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:33.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:33.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:33.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:33.030 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3951 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:33.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:33.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:33.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:33.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:33.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:33.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21cc0285d0 2021-03-09T09:04:33.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0920d0 con 0x7f21cc0285d0 2021-03-09T09:04:33.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09d1a0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f21e4027d50 tx=0x7f21e408e480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:33.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4035380 con 0x7f21cc0285d0 2021-03-09T09:04:33.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.021+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40354e0 con 0x7f21cc0285d0 2021-03-09T09:04:33.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4035380 con 0x7f21cc0285d0 2021-03-09T09:04:33.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.029+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:33.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.029+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:33.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:33.029+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:38.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:38.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:38.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09d1a0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f21e4027d50 tx=0x7f21e408e480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:38.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09d1a0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f21e4027d50 tx=0x7f21e408e480).stop 2021-03-09T09:04:38.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09d1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:38.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09d1a0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:38.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:38.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:38.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:38.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039930 con 0x7f21cc005e30 2021-03-09T09:04:38.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052400 con 0x7f21ec09c280 2021-03-09T09:04:38.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0271e0 con 0x7f21ec0a7f50 2021-03-09T09:04:38.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:38.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:38.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:38.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:38.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:38.029 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3985 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:38.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:38.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:38.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:38.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:38.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc051860 con 0x7f21ec0a7f50 2021-03-09T09:04:38.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:04:38.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09d690 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4035650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:38.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4034580 con 0x7f21ec0a7f50 2021-03-09T09:04:38.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40870e0 con 0x7f21ec0a7f50 2021-03-09T09:04:38.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e070 con 0x7f21ec0a7f50 2021-03-09T09:04:38.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.028+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:38.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.028+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:38.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:38.028+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:43.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:43.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:43.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09d690 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4035650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:43.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09d690 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4035650).stop 2021-03-09T09:04:43.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc09d690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:43.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc09d690 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:43.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:43.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:43.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:43.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f1b0 con 0x7f21cc005e30 2021-03-09T09:04:43.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057cf0 con 0x7f21ec09c280 2021-03-09T09:04:43.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ffd0 con 0x7f21cc0285d0 2021-03-09T09:04:43.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:43.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:43.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:43.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:43.030 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4020 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:43.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:43.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0920d0 con 0x7f21cc0285d0 2021-03-09T09:04:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:04:43.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0919f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:43.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc009120 con 0x7f21cc0285d0 2021-03-09T09:04:43.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21cc0285d0 2021-03-09T09:04:43.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:04:43.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.028+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:43.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:43.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:43.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:47.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:04:47.694 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:04:48.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:48.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:48.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0919f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:48.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0919f0).stop 2021-03-09T09:04:48.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09cd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:48.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:48.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:48.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:48.030 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4058 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:48.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc094020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:48.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a180 con 0x7f21cc005e30 2021-03-09T09:04:48.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013280 con 0x7f21ec09c280 2021-03-09T09:04:48.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a1e0 con 0x7f21ec0a7f50 2021-03-09T09:04:48.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:48.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:48.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:48.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:48.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc094020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:48.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:48.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc014870 con 0x7f21ec0a7f50 2021-03-09T09:04:48.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0920d0 con 0x7f21ec0a7f50 2021-03-09T09:04:48.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc094020 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f21dc068010 tx=0x7f21dc015b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:48.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21ec0a7f50 2021-03-09T09:04:48.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21ec0a7f50 2021-03-09T09:04:48.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc091ca0 con 0x7f21ec0a7f50 2021-03-09T09:04:48.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.032+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:48.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:48.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:48.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:53.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc094020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:53.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc094020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:53.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc094020 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f21dc068010 tx=0x7f21dc015b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:53.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc094020 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f21dc068010 tx=0x7f21dc015b00).stop 2021-03-09T09:04:53.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc094020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:53.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc094020 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:53.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:53.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:53.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:53.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015c70 con 0x7f21cc005e30 2021-03-09T09:04:53.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21ec09c280 2021-03-09T09:04:53.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0271e0 con 0x7f21cc0285d0 2021-03-09T09:04:53.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:53.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:53.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:53.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:53.033 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4092 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:53.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:53.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:53.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:53.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:53.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:53.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0411e0 con 0x7f21cc0285d0 2021-03-09T09:04:53.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc051860 con 0x7f21cc0285d0 2021-03-09T09:04:53.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e405f270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:53.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408e4f0 con 0x7f21cc0285d0 2021-03-09T09:04:53.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0285d0 2021-03-09T09:04:53.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e030 con 0x7f21cc0285d0 2021-03-09T09:04:53.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.032+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:53.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:53.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:53.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:04:58.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:58.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:58.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e405f270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:58.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e405f270).stop 2021-03-09T09:04:58.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09cd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:58.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09cd90 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:58.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:58.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:58.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:58.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08aa90 con 0x7f21cc005e30 2021-03-09T09:04:58.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21ec09c280 2021-03-09T09:04:58.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:04:58.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028fa0 con 0x7f21ec0a7f50 2021-03-09T09:04:58.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:58.035 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:04:58.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:04:58.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:58.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:58.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:58.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:58.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:58.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:58.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057fe0 con 0x7f21ec0a7f50 2021-03-09T09:04:58.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0411e0 con 0x7f21ec0a7f50 2021-03-09T09:04:58.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03b670 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f21cc03cf50 tx=0x7f21e4086650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:58.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4086a50 con 0x7f21ec0a7f50 2021-03-09T09:04:58.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4086bd0 con 0x7f21ec0a7f50 2021-03-09T09:04:58.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e020 con 0x7f21ec0a7f50 2021-03-09T09:04:58.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.036+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:04:58.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:04:58.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:04:58.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:03.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:03.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:03.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03b670 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f21cc03cf50 tx=0x7f21e4086650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:03.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03b670 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f21cc03cf50 tx=0x7f21e4086650).stop 2021-03-09T09:05:03.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03b670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:03.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03b670 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:03.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:03.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:03.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:03.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014ab0 con 0x7f21cc005e30 2021-03-09T09:05:03.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0555a0 con 0x7f21ec09c280 2021-03-09T09:05:03.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08aa90 con 0x7f21cc0285d0 2021-03-09T09:05:03.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:03.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:03.043 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:03.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:03.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:03.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:03.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:03.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ccf0 con 0x7f21cc0285d0 2021-03-09T09:05:03.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc019420 con 0x7f21cc0285d0 2021-03-09T09:05:03.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04d3d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00a6240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:03.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0285d0 2021-03-09T09:05:03.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0285d0 2021-03-09T09:05:03.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0285d0 2021-03-09T09:05:03.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.036+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:03.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.036+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:03.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:03.036+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:08.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:08.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:08.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04d3d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00a6240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:08.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04d3d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00a6240).stop 2021-03-09T09:05:08.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc04d3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:08.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc04d3d0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:08.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:08.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:08.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:08.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0908c0 con 0x7f21cc005e30 2021-03-09T09:05:08.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a180 con 0x7f21ec09c280 2021-03-09T09:05:08.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030390 con 0x7f21ec0a7f50 2021-03-09T09:05:08.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:08.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:08.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:08.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:08.037 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:08.024+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4202 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:08.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:08.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:08.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:08.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:08.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:08.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:05:08.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc014870 con 0x7f21ec0a7f50 2021-03-09T09:05:08.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa90 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f21dc083010 tx=0x7f21dc081280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:08.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053020 con 0x7f21ec0a7f50 2021-03-09T09:05:08.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc053020 con 0x7f21ec0a7f50 2021-03-09T09:05:08.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:05:08.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.036+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:08.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:08.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:08.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:13.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:13.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:13.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa90 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f21dc083010 tx=0x7f21dc081280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:13.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa90 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f21dc083010 tx=0x7f21dc081280).stop 2021-03-09T09:05:13.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03aa90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:13.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa90 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:13.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:13.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:13.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:13.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a50 con 0x7f21cc005e30 2021-03-09T09:05:13.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ab0 con 0x7f21ec09c280 2021-03-09T09:05:13.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03cc10 con 0x7f21cc0285d0 2021-03-09T09:05:13.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:13.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:13.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:13.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:13.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:13.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:13.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:13.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:13.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:13.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:05:13.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21cc0285d0 2021-03-09T09:05:13.040 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4236 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:13.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01d7b0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc010e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:13.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21cc0285d0 2021-03-09T09:05:13.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc0285d0 2021-03-09T09:05:13.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0285d0 2021-03-09T09:05:13.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.032+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:13.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.036+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:13.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:13.036+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:17.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:05:17.749 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:05:18.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:18.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:18.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01d7b0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc010e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:18.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01d7b0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc010e00).stop 2021-03-09T09:05:18.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01d7b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:18.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01d7b0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:18.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:18.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:18.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:18.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08aa90 con 0x7f21cc005e30 2021-03-09T09:05:18.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03cc10 con 0x7f21ec09c280 2021-03-09T09:05:18.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a50 con 0x7f21ec0a7f50 2021-03-09T09:05:18.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:18.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:18.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:18.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:18.040 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4270 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:18.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:18.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:18.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:18.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:18.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:18.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036e10 con 0x7f21ec0a7f50 2021-03-09T09:05:18.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:05:18.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02a9e0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e00a60a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:18.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0088e20 con 0x7f21ec0a7f50 2021-03-09T09:05:18.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:05:18.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:05:18.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.040+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:18.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:18.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:18.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:23.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:23.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:23.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02a9e0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e00a60a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:23.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02a9e0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e00a60a0).stop 2021-03-09T09:05:23.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02a9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:23.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02a9e0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:23.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc02fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:23.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:23.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:23.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02c650 con 0x7f21cc005e30 2021-03-09T09:05:23.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0295c0 con 0x7f21ec09c280 2021-03-09T09:05:23.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097510 con 0x7f21cc0285d0 2021-03-09T09:05:23.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:23.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:23.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:23.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:23.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc02fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:23.047 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4310 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:23.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:23.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:23.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:23.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:23.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040310 con 0x7f21cc0285d0 2021-03-09T09:05:23.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.032+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036e10 con 0x7f21cc0285d0 2021-03-09T09:05:23.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.036+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc02fc30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e00552b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:23.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0055740 con 0x7f21cc0285d0 2021-03-09T09:05:23.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00265d0 con 0x7f21cc0285d0 2021-03-09T09:05:23.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007ccc0 con 0x7f21cc0285d0 2021-03-09T09:05:23.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.040+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:23.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:23.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:23.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:28.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc02fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:28.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc02fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:28.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc02fc30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e00552b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:28.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc02fc30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e00552b0).stop 2021-03-09T09:05:28.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc02fc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:28.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc02fc30 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:28.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:28.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc033420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:28.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:28.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dfa0 con 0x7f21cc005e30 2021-03-09T09:05:28.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae10 con 0x7f21ec09c280 2021-03-09T09:05:28.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023070 con 0x7f21ec0a7f50 2021-03-09T09:05:28.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:28.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:28.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:28.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:28.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc033420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:28.043 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4345 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:28.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:28.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:28.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:28.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:28.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0920d0 con 0x7f21ec0a7f50 2021-03-09T09:05:28.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057fe0 con 0x7f21ec0a7f50 2021-03-09T09:05:28.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc033420 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f21cc044240 tx=0x7f21e4052570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:28.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e405f080 con 0x7f21ec0a7f50 2021-03-09T09:05:28.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009440 con 0x7f21ec0a7f50 2021-03-09T09:05:28.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009440 con 0x7f21ec0a7f50 2021-03-09T09:05:28.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.040+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:28.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:28.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:28.040+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:33.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc033420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:05:33.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc033420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:33.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc033420 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f21cc044240 tx=0x7f21e4052570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:33.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc033420 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f21cc044240 tx=0x7f21e4052570).stop 2021-03-09T09:05:33.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc033420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:33.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc033420 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:33.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:33.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:33.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:33.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c370 con 0x7f21cc005e30 2021-03-09T09:05:33.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae10 con 0x7f21ec09c280 2021-03-09T09:05:33.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f850 con 0x7f21cc0285d0 2021-03-09T09:05:33.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:33.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:33.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:33.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:33.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:33.044 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4376 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:33.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:33.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:33.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:33.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:33.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036e10 con 0x7f21cc0285d0 2021-03-09T09:05:33.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:05:33.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0536a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc077e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:33.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc0285d0 2021-03-09T09:05:33.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc0285d0 2021-03-09T09:05:33.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0285d0 2021-03-09T09:05:33.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.044+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:33.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:33.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:33.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:38.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:05:38.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:38.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0536a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc077e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:38.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0536a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc077e60).stop 2021-03-09T09:05:38.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0536a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:38.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0536a0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:38.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:38.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:38.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:38.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:38.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:38.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:38.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029440 con 0x7f21cc005e30 2021-03-09T09:05:38.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:38.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023070 con 0x7f21ec09c280 2021-03-09T09:05:38.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21ec0a7f50 2021-03-09T09:05:38.046 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4414 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:38.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:38.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:38.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:38.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:38.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:38.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039dd0 con 0x7f21ec0a7f50 2021-03-09T09:05:38.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036e10 con 0x7f21ec0a7f50 2021-03-09T09:05:38.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02be40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f21cc01f9d0 tx=0x7f21dc08b5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:38.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06e900 con 0x7f21ec0a7f50 2021-03-09T09:05:38.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06ea60 con 0x7f21ec0a7f50 2021-03-09T09:05:38.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc059000 con 0x7f21ec0a7f50 2021-03-09T09:05:38.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.043+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:38.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.043+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:38.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:38.043+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:43.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:43.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:43.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02be40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f21cc01f9d0 tx=0x7f21dc08b5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:43.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02be40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f21cc01f9d0 tx=0x7f21dc08b5a0).stop 2021-03-09T09:05:43.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02be40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02be40 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00fd60 con 0x7f21cc005e30 2021-03-09T09:05:43.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023070 con 0x7f21ec09c280 2021-03-09T09:05:43.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027230 con 0x7f21cc0285d0 2021-03-09T09:05:43.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:43.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:43.046 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4453 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:43.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:43.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:43.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:43.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:43.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:43.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:43.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:43.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f300 con 0x7f21cc0285d0 2021-03-09T09:05:43.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039dd0 con 0x7f21cc0285d0 2021-03-09T09:05:43.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc050510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:43.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06e900 con 0x7f21cc0285d0 2021-03-09T09:05:43.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc04fc00 con 0x7f21cc0285d0 2021-03-09T09:05:43.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc091d40 con 0x7f21cc0285d0 2021-03-09T09:05:43.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.043+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:43.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.047+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:43.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:43.047+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:47.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:05:47.803 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:05:48.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:48.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:48.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc050510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:48.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc050510).stop 2021-03-09T09:05:48.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09ad60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:48.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:48.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:48.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:48.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:48.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041e20 con 0x7f21cc005e30 2021-03-09T09:05:48.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08aa90 con 0x7f21ec09c280 2021-03-09T09:05:48.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec0a7f50 2021-03-09T09:05:48.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:48.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:48.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:48.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:48.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:48.048 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:48.039+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4486 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:48.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:48.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:48.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:48.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:48.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:05:48.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0920d0 con 0x7f21ec0a7f50 2021-03-09T09:05:48.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02faf0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f21e40344b0 tx=0x7f21e4009660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:48.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4057750 con 0x7f21ec0a7f50 2021-03-09T09:05:48.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40520d0 con 0x7f21ec0a7f50 2021-03-09T09:05:48.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409e070 con 0x7f21ec0a7f50 2021-03-09T09:05:48.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.047+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:48.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.051+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:48.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:48.051+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:53.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:05:53.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:53.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02faf0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f21e40344b0 tx=0x7f21e4009660).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:53.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02faf0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f21e40344b0 tx=0x7f21e4009660).stop 2021-03-09T09:05:53.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02faf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:53.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02faf0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:53.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:53.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033ec0 con 0x7f21cc005e30 2021-03-09T09:05:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da40 con 0x7f21ec09c280 2021-03-09T09:05:53.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fe50 con 0x7f21cc0285d0 2021-03-09T09:05:53.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:53.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:53.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:53.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:53.050 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4520 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:53.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:53.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:53.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:53.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:53.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:53.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044790 con 0x7f21cc0285d0 2021-03-09T09:05:53.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f300 con 0x7f21cc0285d0 2021-03-09T09:05:53.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01a570 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc0506d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:53.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06e030 con 0x7f21cc0285d0 2021-03-09T09:05:53.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06e030 con 0x7f21cc0285d0 2021-03-09T09:05:53.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc08b250 con 0x7f21cc0285d0 2021-03-09T09:05:53.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.055+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:53.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:53.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:53.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:05:58.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:05:58.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:58.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01a570 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc0506d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:58.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01a570 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc0506d0).stop 2021-03-09T09:05:58.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc01a570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:58.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc01a570 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:58.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:58.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:58.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:58.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013010 con 0x7f21cc005e30 2021-03-09T09:05:58.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21ec09c280 2021-03-09T09:05:58.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0564d0 con 0x7f21ec0a7f50 2021-03-09T09:05:58.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:05:58.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:58.051 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4561 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:05:58.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:05:58.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:05:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040310 con 0x7f21ec0a7f50 2021-03-09T09:05:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02ec60 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00a6890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:58.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:05:58.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21ec0a7f50 2021-03-09T09:05:58.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a6510 con 0x7f21ec0a7f50 2021-03-09T09:05:58.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.055+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:05:58.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:05:58.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:05:58.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:03.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:03.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:03.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02ec60 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00a6890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:03.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02ec60 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00a6890).stop 2021-03-09T09:06:03.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02ec60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02ec60 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:03.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:03.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049e10 con 0x7f21cc005e30 2021-03-09T09:06:03.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027d40 con 0x7f21ec09c280 2021-03-09T09:06:03.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0497b0 con 0x7f21cc0285d0 2021-03-09T09:06:03.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:03.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:03.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:03.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:03.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:03.052 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:03.043+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4595 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:03.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:03.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:03.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:03.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:03.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052880 con 0x7f21cc0285d0 2021-03-09T09:06:03.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044790 con 0x7f21cc0285d0 2021-03-09T09:06:03.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0398b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f21dc07f6e0 tx=0x7f21dc07f520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:03.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05a070 con 0x7f21cc0285d0 2021-03-09T09:06:03.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21cc0285d0 2021-03-09T09:06:03.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06e9b0 con 0x7f21cc0285d0 2021-03-09T09:06:03.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.055+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:03.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:03.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:03.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:08.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:06:08.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:08.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0398b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f21dc07f6e0 tx=0x7f21dc07f520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:08.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0398b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f21dc07f6e0 tx=0x7f21dc07f520).stop 2021-03-09T09:06:08.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc0398b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:08.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc0398b0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:08.051 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4628 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:08.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:08.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:08.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:08.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b5a0 con 0x7f21cc005e30 2021-03-09T09:06:08.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049e10 con 0x7f21ec09c280 2021-03-09T09:06:08.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004020 con 0x7f21ec0a7f50 2021-03-09T09:06:08.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:08.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:08.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:08.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:08.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:08.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:08.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:08.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:08.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:08.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00acb0 con 0x7f21ec0a7f50 2021-03-09T09:06:08.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052880 con 0x7f21ec0a7f50 2021-03-09T09:06:08.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012cd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f21cc027ec0 tx=0x7f21dc062290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:08.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06a830 con 0x7f21ec0a7f50 2021-03-09T09:06:08.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:06:08.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:06:08.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.055+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:08.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.059+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:08.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:08.059+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:13.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:13.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:13.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012cd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f21cc027ec0 tx=0x7f21dc062290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:13.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012cd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f21cc027ec0 tx=0x7f21dc062290).stop 2021-03-09T09:06:13.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:13.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012cd0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:13.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:13.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:13.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc018740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:13.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049e10 con 0x7f21cc005e30 2021-03-09T09:06:13.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dc00 con 0x7f21ec09c280 2021-03-09T09:06:13.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055720 con 0x7f21cc0285d0 2021-03-09T09:06:13.054 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4669 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:13.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:13.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:13.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:13.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:13.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc018740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:13.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:13.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:13.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:13.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:13.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036e10 con 0x7f21cc0285d0 2021-03-09T09:06:13.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00acb0 con 0x7f21cc0285d0 2021-03-09T09:06:13.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc018740 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc06ada0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:13.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02f350 con 0x7f21cc0285d0 2021-03-09T09:06:13.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02f170 con 0x7f21cc0285d0 2021-03-09T09:06:13.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0285d0 2021-03-09T09:06:13.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.059+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:13.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.059+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:13.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:13.059+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:17.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:06:17.851 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:06:18.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc018740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:18.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc018740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:18.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc018740 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc06ada0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:18.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc018740 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc06ada0).stop 2021-03-09T09:06:18.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc018740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:18.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc018740 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:18.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:18.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:18.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:18.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc020310 con 0x7f21cc005e30 2021-03-09T09:06:18.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:18.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091a90 con 0x7f21ec09c280 2021-03-09T09:06:18.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21ec0a7f50 2021-03-09T09:06:18.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.047+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:18.056 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4705 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:18.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:18.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:18.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:18.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:18.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:18.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:18.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:18.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0982e0 con 0x7f21ec0a7f50 2021-03-09T09:06:18.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036e10 con 0x7f21ec0a7f50 2021-03-09T09:06:18.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043d40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f21dc018040 tx=0x7f21dc062290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:18.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0625c0 con 0x7f21ec0a7f50 2021-03-09T09:06:18.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0625c0 con 0x7f21ec0a7f50 2021-03-09T09:06:18.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:06:18.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.059+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:18.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:18.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:18.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:23.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:23.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:23.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043d40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f21dc018040 tx=0x7f21dc062290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:23.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043d40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f21dc018040 tx=0x7f21dc062290).stop 2021-03-09T09:06:23.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:23.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043d40 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:23.061 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4736 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:23.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:23.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc027820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:23.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:23.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0989f0 con 0x7f21cc005e30 2021-03-09T09:06:23.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030350 con 0x7f21ec09c280 2021-03-09T09:06:23.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dc00 con 0x7f21cc0285d0 2021-03-09T09:06:23.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:23.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:23.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:23.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:23.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc027820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:23.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:23.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:23.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:23.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:23.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049dc0 con 0x7f21cc0285d0 2021-03-09T09:06:23.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc0285d0 2021-03-09T09:06:23.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.055+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc027820 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f21e40269f0 tx=0x7f21e4026b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:23.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0285d0 2021-03-09T09:06:23.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0285d0 2021-03-09T09:06:23.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0285d0 2021-03-09T09:06:23.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.063+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:23.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:23.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:23.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:28.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc027820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:28.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc027820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:28.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc027820 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f21e40269f0 tx=0x7f21e4026b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:28.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc027820 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f21e40269f0 tx=0x7f21e4026b40).stop 2021-03-09T09:06:28.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc027820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:28.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc027820 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:28.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:28.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:28.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:28.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0989f0 con 0x7f21cc005e30 2021-03-09T09:06:28.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7a0 con 0x7f21ec09c280 2021-03-09T09:06:28.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a470 con 0x7f21ec0a7f50 2021-03-09T09:06:28.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:28.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:28.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:28.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:28.060 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4775 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:28.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:28.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:28.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:28.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:28.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:28.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00fd10 con 0x7f21ec0a7f50 2021-03-09T09:06:28.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0982e0 con 0x7f21ec0a7f50 2021-03-09T09:06:28.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018740 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f21cc01dc30 tx=0x7f21dc06ae80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:28.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f540 con 0x7f21ec0a7f50 2021-03-09T09:06:28.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f540 con 0x7f21ec0a7f50 2021-03-09T09:06:28.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:06:28.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.063+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:28.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:28.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:28.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:33.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc018740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:33.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc018740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:33.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018740 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f21cc01dc30 tx=0x7f21dc06ae80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:33.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018740 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f21cc01dc30 tx=0x7f21dc06ae80).stop 2021-03-09T09:06:33.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc018740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:33.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018740 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:33.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:33.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:33.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:33.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:33.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:33.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dc00 con 0x7f21cc005e30 2021-03-09T09:06:33.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f4d0 con 0x7f21ec09c280 2021-03-09T09:06:33.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e600 con 0x7f21cc0285d0 2021-03-09T09:06:33.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:33.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:33.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:33.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:33.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:33.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:33.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:33.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052880 con 0x7f21cc0285d0 2021-03-09T09:06:33.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00fd10 con 0x7f21cc0285d0 2021-03-09T09:06:33.063 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4815 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:33.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043d40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc085800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:33.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f540 con 0x7f21cc0285d0 2021-03-09T09:06:33.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21cc0285d0 2021-03-09T09:06:33.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085c90 con 0x7f21cc0285d0 2021-03-09T09:06:33.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.059+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:33.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.059+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:33.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:33.059+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:38.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc043d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:38.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc043d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:38.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043d40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc085800).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:38.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043d40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f21dc003fd0 tx=0x7f21dc085800).stop 2021-03-09T09:06:38.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc043d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:38.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc043d40 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:38.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:38.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:38.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:38.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc005e30 2021-03-09T09:06:38.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dfa0 con 0x7f21ec09c280 2021-03-09T09:06:38.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a1c0 con 0x7f21ec0a7f50 2021-03-09T09:06:38.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:38.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:38.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:38.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:38.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:38.065 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:38.054+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4849 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:38.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:38.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:38.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:38.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:38.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:06:38.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e009ce70 con 0x7f21ec0a7f50 2021-03-09T09:06:38.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014370 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e00a5fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:38.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21ec0a7f50 2021-03-09T09:06:38.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:06:38.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:06:38.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.066+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:38.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.066+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:38.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:38.066+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:43.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc014370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:43.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc014370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:43.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014370 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e00a5fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:43.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014370 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e00a5fb0).stop 2021-03-09T09:06:43.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc014370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:43.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014370 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:43.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:43.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:43.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:43.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da40 con 0x7f21cc005e30 2021-03-09T09:06:43.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030310 con 0x7f21ec09c280 2021-03-09T09:06:43.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007ba0 con 0x7f21cc0285d0 2021-03-09T09:06:43.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:43.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:43.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:43.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:43.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.054+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:43.064 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4880 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:43.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:43.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:43.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:43.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:43.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f300 con 0x7f21cc0285d0 2021-03-09T09:06:43.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21cc0285d0 2021-03-09T09:06:43.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c6e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f21e009c360 tx=0x7f21e0058cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:43.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6720 con 0x7f21cc0285d0 2021-03-09T09:06:43.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00982f0 con 0x7f21cc0285d0 2021-03-09T09:06:43.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00580d0 con 0x7f21cc0285d0 2021-03-09T09:06:43.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.062+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:43.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.062+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:43.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:43.062+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:47.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:06:47.890 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:06:48.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:48.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:48.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c6e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f21e009c360 tx=0x7f21e0058cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:48.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c6e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f21e009c360 tx=0x7f21e0058cf0).stop 2021-03-09T09:06:48.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:48.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c6e0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:48.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:48.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:48.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:48.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f40 con 0x7f21cc005e30 2021-03-09T09:06:48.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0475c0 con 0x7f21ec09c280 2021-03-09T09:06:48.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21ec0a7f50 2021-03-09T09:06:48.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:48.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:48.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:48.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:48.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:48.066 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4916 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:48.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:48.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:48.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:48.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:48.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:06:48.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052880 con 0x7f21ec0a7f50 2021-03-09T09:06:48.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.062+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02abb0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f21dc093010 tx=0x7f21dc015b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:48.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053070 con 0x7f21ec0a7f50 2021-03-09T09:06:48.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc04e9b0 con 0x7f21ec0a7f50 2021-03-09T09:06:48.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:06:48.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.066+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:48.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:48.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:48.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:53.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:53.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:53.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02abb0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f21dc093010 tx=0x7f21dc015b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:53.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02abb0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f21dc093010 tx=0x7f21dc015b30).stop 2021-03-09T09:06:53.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02abb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:53.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02abb0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:53.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:53.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:53.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:53.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a840 con 0x7f21cc005e30 2021-03-09T09:06:53.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091e80 con 0x7f21ec09c280 2021-03-09T09:06:53.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21cc0285d0 2021-03-09T09:06:53.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:53.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:53.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:53.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:53.068 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4952 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:53.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.058+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:53.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:53.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:53.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:53.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:53.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21cc0285d0 2021-03-09T09:06:53.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049dc0 con 0x7f21cc0285d0 2021-03-09T09:06:53.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c520 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f21e4085e10 tx=0x7f21e4085d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:53.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089020 con 0x7f21cc0285d0 2021-03-09T09:06:53.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0285d0 2021-03-09T09:06:53.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21cc0285d0 2021-03-09T09:06:53.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.070+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:53.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:53.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:53.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:06:58.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:06:58.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:58.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c520 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f21e4085e10 tx=0x7f21e4085d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:58.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c520 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f21e4085e10 tx=0x7f21e4085d40).stop 2021-03-09T09:06:58.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:58.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c520 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:58.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:58.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:58.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:58.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21cc005e30 2021-03-09T09:06:58.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d80 con 0x7f21ec09c280 2021-03-09T09:06:58.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21ec0a7f50 2021-03-09T09:06:58.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:06:58.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:58.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:06:58.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:58.069 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4992 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:06:58.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:58.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:58.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:58.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:58.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:58.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0221c0 con 0x7f21ec0a7f50 2021-03-09T09:06:58.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:06:58.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026bf0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f21cc02bd40 tx=0x7f21e40aa430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:58.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T09:06:58.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:06:58.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T09:06:58.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.070+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:06:58.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.074+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:06:58.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:06:58.074+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:03.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:03.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:03.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026bf0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f21cc02bd40 tx=0x7f21e40aa430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:03.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026bf0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f21cc02bd40 tx=0x7f21e40aa430).stop 2021-03-09T09:07:03.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:03.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026bf0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:03.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:03.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:03.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:03.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02c890 con 0x7f21cc005e30 2021-03-09T09:07:03.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0301d0 con 0x7f21ec09c280 2021-03-09T09:07:03.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ade0 con 0x7f21cc0285d0 2021-03-09T09:07:03.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:03.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:03.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:03.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:03.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:03.070 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5031 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:03.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:03.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:03.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:03.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:03.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013820 con 0x7f21cc0285d0 2021-03-09T09:07:03.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0221c0 con 0x7f21cc0285d0 2021-03-09T09:07:03.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c1c0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40571b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:03.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4052360 con 0x7f21cc0285d0 2021-03-09T09:07:03.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40524c0 con 0x7f21cc0285d0 2021-03-09T09:07:03.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4035170 con 0x7f21cc0285d0 2021-03-09T09:07:03.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.066+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:03.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:03.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:03.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:08.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:08.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:08.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c1c0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40571b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:08.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c1c0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40571b0).stop 2021-03-09T09:07:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03c1c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03c1c0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da40 con 0x7f21cc005e30 2021-03-09T09:07:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21ec09c280 2021-03-09T09:07:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024680 con 0x7f21ec0a7f50 2021-03-09T09:07:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.062+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:08.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:08.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:08.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:08.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08a170 con 0x7f21ec0a7f50 2021-03-09T09:07:08.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f300 con 0x7f21ec0a7f50 2021-03-09T09:07:08.074 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5060 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:08.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098d50 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:08.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21ec0a7f50 2021-03-09T09:07:08.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:07:08.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009c000 con 0x7f21ec0a7f50 2021-03-09T09:07:08.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.070+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:08.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:08.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:08.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:13.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc098d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:13.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc098d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:13.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098d50 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:13.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098d50 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003740).stop 2021-03-09T09:07:13.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc098d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:13.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc098d50 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:13.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:13.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:13.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:13.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da40 con 0x7f21cc005e30 2021-03-09T09:07:13.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04caf0 con 0x7f21ec09c280 2021-03-09T09:07:13.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0290c0 con 0x7f21cc0285d0 2021-03-09T09:07:13.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:13.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:13.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:13.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:13.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:13.073 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5103 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:13.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:13.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:13.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:13.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:13.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f300 con 0x7f21cc0285d0 2021-03-09T09:07:13.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013820 con 0x7f21cc0285d0 2021-03-09T09:07:13.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03b4f0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f21e4087990 tx=0x7f21e40ac080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:13.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21cc0285d0 2021-03-09T09:07:13.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40526d0 con 0x7f21cc0285d0 2021-03-09T09:07:13.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40520a0 con 0x7f21cc0285d0 2021-03-09T09:07:13.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.074+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:13.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.074+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:13.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:13.074+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:17.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:07:17.928 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:07:18.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:18.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:18.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03b4f0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f21e4087990 tx=0x7f21e40ac080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:18.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03b4f0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f21e4087990 tx=0x7f21e40ac080).stop 2021-03-09T09:07:18.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:18.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc03b4f0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:18.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:18.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:18.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:18.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036030 con 0x7f21cc005e30 2021-03-09T09:07:18.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022b10 con 0x7f21ec09c280 2021-03-09T09:07:18.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da40 con 0x7f21ec0a7f50 2021-03-09T09:07:18.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:18.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:18.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:18.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:18.076 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:18.066+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5137 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:18.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:18.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:18.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:18.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:18.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:18.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012b70 con 0x7f21ec0a7f50 2021-03-09T09:07:18.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08a170 con 0x7f21ec0a7f50 2021-03-09T09:07:18.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e00783b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:18.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21ec0a7f50 2021-03-09T09:07:18.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21ec0a7f50 2021-03-09T09:07:18.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009c000 con 0x7f21ec0a7f50 2021-03-09T09:07:18.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.078+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:18.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.078+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:18.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:18.078+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:23.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:23.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:23.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e00783b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:23.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e00783b0).stop 2021-03-09T09:07:23.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:23.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:23.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc039ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:23.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:23.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:23.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0303e0 con 0x7f21cc005e30 2021-03-09T09:07:23.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a6b0 con 0x7f21ec09c280 2021-03-09T09:07:23.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097330 con 0x7f21cc0285d0 2021-03-09T09:07:23.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:23.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:23.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:23.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:23.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc039ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:23.077 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5168 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:23.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:23.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:23.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:23.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:23.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053100 con 0x7f21cc0285d0 2021-03-09T09:07:23.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b70 con 0x7f21cc0285d0 2021-03-09T09:07:23.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.074+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc039ad0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f21e00373c0 tx=0x7f21e0036e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:23.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0285d0 2021-03-09T09:07:23.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0285d0 2021-03-09T09:07:23.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.082+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0285d0 2021-03-09T09:07:23.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.082+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:23.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.082+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:23.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:23.082+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:28.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc039ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:28.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc039ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:28.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc039ad0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f21e00373c0 tx=0x7f21e0036e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:28.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc039ad0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f21e00373c0 tx=0x7f21e0036e60).stop 2021-03-09T09:07:28.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc039ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:28.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc039ad0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:28.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:28.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:28.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:28.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0290c0 con 0x7f21cc005e30 2021-03-09T09:07:28.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a9f0 con 0x7f21ec09c280 2021-03-09T09:07:28.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036370 con 0x7f21ec0a7f50 2021-03-09T09:07:28.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:28.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:28.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:28.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:28.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:28.079 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:28.070+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:28.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:28.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:28.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:28.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:28.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0492d0 con 0x7f21ec0a7f50 2021-03-09T09:07:28.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f300 con 0x7f21ec0a7f50 2021-03-09T09:07:28.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa40 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f21cc034560 tx=0x7f21e4051a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:28.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:07:28.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40abcf0 con 0x7f21ec0a7f50 2021-03-09T09:07:28.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4052040 con 0x7f21ec0a7f50 2021-03-09T09:07:28.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.078+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:28.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.078+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:28.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:28.078+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:33.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.070+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:33.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.070+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:33.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.070+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa40 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f21cc034560 tx=0x7f21e4051a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:33.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.070+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa40 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f21cc034560 tx=0x7f21e4051a30).stop 2021-03-09T09:07:33.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.070+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03aa40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:33.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.070+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03aa40 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:33.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:33.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:33.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:33.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036370 con 0x7f21cc005e30 2021-03-09T09:07:33.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028490 con 0x7f21ec09c280 2021-03-09T09:07:33.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ff00 con 0x7f21cc0285d0 2021-03-09T09:07:33.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:33.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:33.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:33.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:33.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:33.081 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5247 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:33.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 msgr2=0x7f21cc09ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:07:33.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0285d0 0x7f21cc09ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:33.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 msgr2=0x7f21cc08af80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:33.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21ec09c280 0x7f21cc08af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:33.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:33.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:33.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21c40057f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:33.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21c4008230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:33.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c400a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:33.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036370 con 0x7f21cc005e30 2021-03-09T09:07:33.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028490 con 0x7f21c4005da0 2021-03-09T09:07:33.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ff00 con 0x7f21c40033c0 2021-03-09T09:07:33.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21c4008230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:33.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21c4008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:33.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21c400a870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:33.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.074+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c400a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:33.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.078+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21c40057f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:33.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21c4008230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:33.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21c4008230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:33.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21c400a870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:33.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c400a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:33.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08f760 con 0x7f21c40033c0 2021-03-09T09:07:33.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c40033c0 2021-03-09T09:07:33.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.082+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21c40057f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007e500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:33.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c40033c0 2021-03-09T09:07:33.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c40033c0 2021-03-09T09:07:33.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00783b0 con 0x7f21c40033c0 2021-03-09T09:07:33.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.086+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:33.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.086+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:33.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:33.086+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:38.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21c40057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:38.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21c40057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:38.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21c40057f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007e500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:38.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21c40057f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007e500).stop 2021-03-09T09:07:38.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21c40057f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:38.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21c40057f0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:38.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:38.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:38.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:38.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0290c0 con 0x7f21cc005e30 2021-03-09T09:07:38.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019140 con 0x7f21c4005da0 2021-03-09T09:07:38.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036370 con 0x7f21ec0a7f50 2021-03-09T09:07:38.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:38.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:38.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:38.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:38.083 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5281 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:38.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:38.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:38.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:38.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:38.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:38.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f300 con 0x7f21ec0a7f50 2021-03-09T09:07:38.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08f760 con 0x7f21ec0a7f50 2021-03-09T09:07:38.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023e00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e00a6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:38.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00786a0 con 0x7f21ec0a7f50 2021-03-09T09:07:38.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d1d0 con 0x7f21ec0a7f50 2021-03-09T09:07:38.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00783b0 con 0x7f21ec0a7f50 2021-03-09T09:07:38.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.081+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:38.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:38.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:38.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:43.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc023e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:43.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc023e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:43.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023e00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e00a6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:43.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023e00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e00a6200).stop 2021-03-09T09:07:43.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc023e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:43.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc023e00 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:43.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:43.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:43.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc013a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:43.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028490 con 0x7f21cc005e30 2021-03-09T09:07:43.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016be0 con 0x7f21c4005da0 2021-03-09T09:07:43.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036370 con 0x7f21c40033c0 2021-03-09T09:07:43.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:43.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:43.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc013a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:43.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:43.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:43.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:43.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:43.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:43.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:43.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21c40033c0 2021-03-09T09:07:43.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c40033c0 2021-03-09T09:07:43.085 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:43.073+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5319 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:43.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc013a70 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc010f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:43.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:07:43.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc090440 con 0x7f21c40033c0 2021-03-09T09:07:43.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc090440 con 0x7f21c40033c0 2021-03-09T09:07:43.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.081+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:43.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:43.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:43.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:47.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:07:47.975 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:07:48.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc013a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:48.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc013a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:48.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc013a70 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc010f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:48.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc013a70 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc010f30).stop 2021-03-09T09:07:48.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc013a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:48.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc013a70 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:48.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:48.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:48.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:48.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0101e0 con 0x7f21cc005e30 2021-03-09T09:07:48.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f6e0 con 0x7f21c4005da0 2021-03-09T09:07:48.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0338e0 con 0x7f21ec0a7f50 2021-03-09T09:07:48.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:48.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:48.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:48.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:48.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:48.088 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5347 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:48.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:48.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:48.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:48.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:48.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08a170 con 0x7f21ec0a7f50 2021-03-09T09:07:48.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0492d0 con 0x7f21ec0a7f50 2021-03-09T09:07:48.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f8d0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7f21e4053f50 tx=0x7f21e4087d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:48.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:07:48.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009590 con 0x7f21ec0a7f50 2021-03-09T09:07:48.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4035170 con 0x7f21ec0a7f50 2021-03-09T09:07:48.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.085+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:48.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:48.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:48.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:53.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:53.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:53.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f8d0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7f21e4053f50 tx=0x7f21e4087d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:53.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f8d0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7f21e4053f50 tx=0x7f21e4087d80).stop 2021-03-09T09:07:53.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:53.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f8d0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:53.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:53.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:53.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:53.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a800 con 0x7f21cc005e30 2021-03-09T09:07:53.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d8d0 con 0x7f21c4005da0 2021-03-09T09:07:53.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041da0 con 0x7f21c40033c0 2021-03-09T09:07:53.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:53.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:53.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:53.086 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5387 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:53.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:53.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.077+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:53.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:53.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:53.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:53.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:53.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21c40033c0 2021-03-09T09:07:53.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21c40033c0 2021-03-09T09:07:53.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f21cc056060 tx=0x7f21dc069500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:53.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053070 con 0x7f21c40033c0 2021-03-09T09:07:53.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21c40033c0 2021-03-09T09:07:53.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ad2a0 con 0x7f21c40033c0 2021-03-09T09:07:53.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.085+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:53.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:53.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:53.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:07:58.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:58.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:58.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f21cc056060 tx=0x7f21dc069500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:58.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f21cc056060 tx=0x7f21dc069500).stop 2021-03-09T09:07:58.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:58.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:58.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:58.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0901b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:58.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:58.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028490 con 0x7f21cc005e30 2021-03-09T09:07:58.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033400 con 0x7f21c4005da0 2021-03-09T09:07:58.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016be0 con 0x7f21ec0a7f50 2021-03-09T09:07:58.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0901b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:07:58.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:07:58.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0901b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:58.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:58.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:58.092 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:07:58.077+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5423 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:07:58.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:58.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:58.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0901b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:58.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0901b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:58.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013820 con 0x7f21ec0a7f50 2021-03-09T09:07:58.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f300 con 0x7f21ec0a7f50 2021-03-09T09:07:58.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018c20 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e007e500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:58.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0037200 con 0x7f21ec0a7f50 2021-03-09T09:07:58.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:07:58.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:07:58.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.089+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:07:58.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:07:58.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:07:58.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:03.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc018c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:03.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc018c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:03.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018c20 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e007e500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:03.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018c20 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e007e500).stop 2021-03-09T09:08:03.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc018c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:03.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc018c20 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:03.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:03.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:03.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:03.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0290c0 con 0x7f21cc005e30 2021-03-09T09:08:03.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022700 con 0x7f21c4005da0 2021-03-09T09:08:03.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d910 con 0x7f21c40033c0 2021-03-09T09:08:03.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:03.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:03.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:03.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:03.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:03.090 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5458 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:03.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:03.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:03.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:03.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:03.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012b70 con 0x7f21c40033c0 2021-03-09T09:08:03.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013820 con 0x7f21c40033c0 2021-03-09T09:08:03.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:03.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d2a0 con 0x7f21c40033c0 2021-03-09T09:08:03.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d2a0 con 0x7f21c40033c0 2021-03-09T09:08:03.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0087170 con 0x7f21c40033c0 2021-03-09T09:08:03.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.089+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:03.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:03.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:03.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:08.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:08.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:08.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:08.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007c5a0).stop 2021-03-09T09:08:08.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:08.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:08.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:08.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:08.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:08.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095b30 con 0x7f21cc005e30 2021-03-09T09:08:08.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0338e0 con 0x7f21c4005da0 2021-03-09T09:08:08.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011800 con 0x7f21ec0a7f50 2021-03-09T09:08:08.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:08.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:08.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:08.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:08.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:08.091 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:08.081+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5495 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:08.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:08.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:08.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc08d460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:08.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc08d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:08.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:08:08.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21ec0a7f50 2021-03-09T09:08:08.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f21dc080010 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:08.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053030 con 0x7f21ec0a7f50 2021-03-09T09:08:08.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07f960 con 0x7f21ec0a7f50 2021-03-09T09:08:08.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21ec0a7f50 2021-03-09T09:08:08.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.089+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:08.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:08.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:08.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:13.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:13.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:13.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f21dc080010 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:13.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f21dc080010 tx=0x7f21dc077e50).stop 2021-03-09T09:08:13.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c7a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:13.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c7a0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:13.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:13.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:13.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:13.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030300 con 0x7f21cc005e30 2021-03-09T09:08:13.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091640 con 0x7f21c4005da0 2021-03-09T09:08:13.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d680 con 0x7f21c40033c0 2021-03-09T09:08:13.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:13.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:13.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:13.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:13.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:13.093 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5531 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:13.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:13.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:13.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:13.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:13.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0556e0 con 0x7f21c40033c0 2021-03-09T09:08:13.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08a170 con 0x7f21c40033c0 2021-03-09T09:08:13.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f21cc02b280 tx=0x7f21e40a17f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:13.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21c40033c0 2021-03-09T09:08:13.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409cd00 con 0x7f21c40033c0 2021-03-09T09:08:13.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4035170 con 0x7f21c40033c0 2021-03-09T09:08:13.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.093+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:13.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:13.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:13.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:18.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:08:18.024 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:08:18.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:18.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:18.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f21cc02b280 tx=0x7f21e40a17f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f21cc02b280 tx=0x7f21e40a17f0).stop 2021-03-09T09:08:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:18.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:18.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:18.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0290c0 con 0x7f21cc005e30 2021-03-09T09:08:18.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21c4005da0 2021-03-09T09:08:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0472c0 con 0x7f21ec0a7f50 2021-03-09T09:08:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:18.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:18.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:18.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:18.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:18.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:18.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04b460 con 0x7f21ec0a7f50 2021-03-09T09:08:18.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0556e0 con 0x7f21ec0a7f50 2021-03-09T09:08:18.097 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:18.085+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5568 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:18.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f21e407e040 tx=0x7f21e40abab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:18.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:08:18.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40ab6a0 con 0x7f21ec0a7f50 2021-03-09T09:08:18.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T09:08:18.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.093+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:18.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:18.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:18.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:23.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:23.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:23.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f21e407e040 tx=0x7f21e40abab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f21e407e040 tx=0x7f21e40abab0).stop 2021-03-09T09:08:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566a0 con 0x7f21cc005e30 2021-03-09T09:08:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21c4005da0 2021-03-09T09:08:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091ec0 con 0x7f21c40033c0 2021-03-09T09:08:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:23.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:23.097 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5609 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:23.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:23.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f140 con 0x7f21c40033c0 2021-03-09T09:08:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04b460 con 0x7f21c40033c0 2021-03-09T09:08:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f21e40aba40 tx=0x7f21e400da00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21c40033c0 2021-03-09T09:08:23.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400ddd0 con 0x7f21c40033c0 2021-03-09T09:08:23.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016080 con 0x7f21c40033c0 2021-03-09T09:08:23.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.097+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:23.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:23.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:23.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:28.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:28.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:28.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f21e40aba40 tx=0x7f21e400da00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:28.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f21e40aba40 tx=0x7f21e400da00).stop 2021-03-09T09:08:28.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:28.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:28.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:28.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:28.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:28.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009560 con 0x7f21cc005e30 2021-03-09T09:08:28.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0181a0 con 0x7f21c4005da0 2021-03-09T09:08:28.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d7b0 con 0x7f21ec0a7f50 2021-03-09T09:08:28.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:28.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:28.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:28.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:28.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:28.099 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5643 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:28.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:28.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:28.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:28.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:28.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053100 con 0x7f21ec0a7f50 2021-03-09T09:08:28.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f140 con 0x7f21ec0a7f50 2021-03-09T09:08:28.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c780 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f21e407ded0 tx=0x7f21e40ab550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:28.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21ec0a7f50 2021-03-09T09:08:28.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4034e70 con 0x7f21ec0a7f50 2021-03-09T09:08:28.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016080 con 0x7f21ec0a7f50 2021-03-09T09:08:28.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.097+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:28.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:33.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:33.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:33.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c780 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f21e407ded0 tx=0x7f21e40ab550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:33.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c780 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f21e407ded0 tx=0x7f21e40ab550).stop 2021-03-09T09:08:33.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:33.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c780 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:33.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:33.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:33.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:33.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0290c0 con 0x7f21cc005e30 2021-03-09T09:08:33.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d910 con 0x7f21c4005da0 2021-03-09T09:08:33.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21c40033c0 2021-03-09T09:08:33.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:33.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:33.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:33.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:33.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:33.102 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:33.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:33.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:33.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:33.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:33.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0492d0 con 0x7f21c40033c0 2021-03-09T09:08:33.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c40033c0 2021-03-09T09:08:33.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f21dc091220 tx=0x7f21dc07fb40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:33.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053020 con 0x7f21c40033c0 2021-03-09T09:08:33.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c40033c0 2021-03-09T09:08:33.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0203a0 con 0x7f21c40033c0 2021-03-09T09:08:33.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.101+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:33.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:33.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:33.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:38.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:38.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:38.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f21dc091220 tx=0x7f21dc07fb40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:38.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f21dc091220 tx=0x7f21dc07fb40).stop 2021-03-09T09:08:38.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:38.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:38.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:38.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:38.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:38.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096d10 con 0x7f21cc005e30 2021-03-09T09:08:38.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047600 con 0x7f21c4005da0 2021-03-09T09:08:38.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566a0 con 0x7f21ec0a7f50 2021-03-09T09:08:38.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:38.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:38.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:38.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:38.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:38.102 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5714 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:38.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:38.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:38.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:38.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:38.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:08:38.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.092+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b70 con 0x7f21ec0a7f50 2021-03-09T09:08:38.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.096+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e00090c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:38.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000b3e0 con 0x7f21ec0a7f50 2021-03-09T09:08:38.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6200 con 0x7f21ec0a7f50 2021-03-09T09:08:38.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0087a40 con 0x7f21ec0a7f50 2021-03-09T09:08:38.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.104+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:38.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:38.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:38.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:43.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:43.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:43.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e00090c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:43.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e00090c0).stop 2021-03-09T09:08:43.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:43.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c5a0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:43.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:43.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:43.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:43.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:43.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a840 con 0x7f21cc005e30 2021-03-09T09:08:43.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022700 con 0x7f21c4005da0 2021-03-09T09:08:43.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:43.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21c40033c0 2021-03-09T09:08:43.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:43.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:43.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:43.103 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5759 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:43.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:43.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:43.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:43.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:43.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057580 con 0x7f21c40033c0 2021-03-09T09:08:43.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053100 con 0x7f21c40033c0 2021-03-09T09:08:43.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7f21cc044be0 tx=0x7f21e4033890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:43.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21c40033c0 2021-03-09T09:08:43.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4033c00 con 0x7f21c40033c0 2021-03-09T09:08:43.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40ab5c0 con 0x7f21c40033c0 2021-03-09T09:08:43.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.100+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:43.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.100+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:43.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:43.100+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:48.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:08:48.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:48.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:48.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7f21cc044be0 tx=0x7f21e4033890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:48.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7f21cc044be0 tx=0x7f21e4033890).stop 2021-03-09T09:08:48.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:48.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:48.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:48.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0223c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:48.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:48.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ade0 con 0x7f21cc005e30 2021-03-09T09:08:48.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:48.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ac20 con 0x7f21c4005da0 2021-03-09T09:08:48.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21ec0a7f50 2021-03-09T09:08:48.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:48.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:48.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:48.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0223c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:48.105 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:08:48.106 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5787 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:48.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:48.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:48.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:48.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:48.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02c140 con 0x7f21ec0a7f50 2021-03-09T09:08:48.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057580 con 0x7f21ec0a7f50 2021-03-09T09:08:48.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0223c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f21e40a1ad0 tx=0x7f21e4005e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:48.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:08:48.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400dc80 con 0x7f21ec0a7f50 2021-03-09T09:08:48.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400dc80 con 0x7f21ec0a7f50 2021-03-09T09:08:48.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.104+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:48.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:48.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:48.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:53.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0223c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:53.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0223c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:53.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0223c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f21e40a1ad0 tx=0x7f21e4005e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:53.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0223c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f21e40a1ad0 tx=0x7f21e4005e90).stop 2021-03-09T09:08:53.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0223c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:53.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0223c0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:53.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:53.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:53.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:53.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566a0 con 0x7f21cc005e30 2021-03-09T09:08:53.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036370 con 0x7f21c4005da0 2021-03-09T09:08:53.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a840 con 0x7f21c40033c0 2021-03-09T09:08:53.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:53.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:53.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:53.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:53.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:53.112 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5822 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:53.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:53.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:53.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:53.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:53.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0368a0 con 0x7f21c40033c0 2021-03-09T09:08:53.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0492d0 con 0x7f21c40033c0 2021-03-09T09:08:53.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:53.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0278a0 con 0x7f21c40033c0 2021-03-09T09:08:53.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ad280 con 0x7f21c40033c0 2021-03-09T09:08:53.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21c40033c0 2021-03-09T09:08:53.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.108+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:53.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:53.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:53.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:08:58.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:08:58.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:58.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:58.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07ff90).stop 2021-03-09T09:08:58.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:58.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:58.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:58.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:58.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc037570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:58.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dd30 con 0x7f21cc005e30 2021-03-09T09:08:58.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0384d0 con 0x7f21c4005da0 2021-03-09T09:08:58.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028350 con 0x7f21ec0a7f50 2021-03-09T09:08:58.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:08:58.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:58.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:08:58.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:58.109 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5854 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:08:58.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc037570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:58.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:58.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:58.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:58.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:58.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049de0 con 0x7f21ec0a7f50 2021-03-09T09:08:58.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0368a0 con 0x7f21ec0a7f50 2021-03-09T09:08:58.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc037570 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc06e860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:58.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ad4e0 con 0x7f21ec0a7f50 2021-03-09T09:08:58.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ad640 con 0x7f21ec0a7f50 2021-03-09T09:08:58.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21ec0a7f50 2021-03-09T09:08:58.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.108+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:08:58.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:08:58.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:08:58.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:03.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc037570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:03.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc037570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:03.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc037570 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc06e860).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:03.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc037570 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc06e860).stop 2021-03-09T09:09:03.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc037570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:03.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc037570 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:03.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:03.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc053d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:03.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:03.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04bb30 con 0x7f21cc005e30 2021-03-09T09:09:03.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b080 con 0x7f21c4005da0 2021-03-09T09:09:03.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055730 con 0x7f21c40033c0 2021-03-09T09:09:03.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:03.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:03.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:03.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:03.110 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5896 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:03.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc053d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:03.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:03.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:03.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:03.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:03.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057580 con 0x7f21c40033c0 2021-03-09T09:09:03.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02c140 con 0x7f21c40033c0 2021-03-09T09:09:03.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc053d30 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f21e4018000 tx=0x7f21e40a38a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:03.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21c40033c0 2021-03-09T09:09:03.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4035240 con 0x7f21c40033c0 2021-03-09T09:09:03.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c40033c0 2021-03-09T09:09:03.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.112+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:03.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:03.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:03.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:08.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc053d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:08.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc053d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:08.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc053d30 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f21e4018000 tx=0x7f21e40a38a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:08.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc053d30 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f21e4018000 tx=0x7f21e40a38a0).stop 2021-03-09T09:09:08.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc053d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:08.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc053d30 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:08.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:08.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:08.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:08.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ad0 con 0x7f21cc005e30 2021-03-09T09:09:08.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045030 con 0x7f21c4005da0 2021-03-09T09:09:08.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:08.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0402d0 con 0x7f21ec0a7f50 2021-03-09T09:09:08.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:08.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:08.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:08.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:08.113 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5929 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:08.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:08.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:08.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:08.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:08.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012b70 con 0x7f21ec0a7f50 2021-03-09T09:09:08.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049de0 con 0x7f21ec0a7f50 2021-03-09T09:09:08.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043400 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7f21cc04e3d0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:08.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02fe60 con 0x7f21ec0a7f50 2021-03-09T09:09:08.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ad280 con 0x7f21ec0a7f50 2021-03-09T09:09:08.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02fad0 con 0x7f21ec0a7f50 2021-03-09T09:09:08.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.112+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:08.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.112+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:08.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:08.112+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:13.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:09:13.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:13.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043400 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7f21cc04e3d0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:13.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043400 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7f21cc04e3d0 tx=0x7f21dc077de0).stop 2021-03-09T09:09:13.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:13.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043400 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:13.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:13.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:13.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:13.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019740 con 0x7f21cc005e30 2021-03-09T09:09:13.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053ba0 con 0x7f21c4005da0 2021-03-09T09:09:13.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ab0 con 0x7f21c40033c0 2021-03-09T09:09:13.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:13.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:13.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:13.114 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5966 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:13.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:13.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:13.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:13.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:13.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:13.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:13.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02e330 con 0x7f21c40033c0 2021-03-09T09:09:13.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b70 con 0x7f21c40033c0 2021-03-09T09:09:13.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c560 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f21dc0541c0 tx=0x7f21dc0ad280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:13.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02fad0 con 0x7f21c40033c0 2021-03-09T09:09:13.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02fc30 con 0x7f21c40033c0 2021-03-09T09:09:13.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21c40033c0 2021-03-09T09:09:13.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.112+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:13.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.112+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:13.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:13.112+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:18.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:18.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:18.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c560 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f21dc0541c0 tx=0x7f21dc0ad280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:18.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c560 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f21dc0541c0 tx=0x7f21dc0ad280).stop 2021-03-09T09:09:18.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:18.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c560 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:18.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:18.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:18.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:18.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004540 con 0x7f21cc005e30 2021-03-09T09:09:18.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d250 con 0x7f21c4005da0 2021-03-09T09:09:18.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036200 con 0x7f21ec0a7f50 2021-03-09T09:09:18.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:18.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.104+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:18.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:18.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:18.116 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6000 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:18.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:18.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:18.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:18.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:18.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:18.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00e400 con 0x7f21ec0a7f50 2021-03-09T09:09:18.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057580 con 0x7f21ec0a7f50 2021-03-09T09:09:18.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043060 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f21cc036ee0 tx=0x7f21e40352d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:18.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089040 con 0x7f21ec0a7f50 2021-03-09T09:09:18.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407fb80 con 0x7f21ec0a7f50 2021-03-09T09:09:18.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21ec0a7f50 2021-03-09T09:09:18.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.116+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:18.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:18.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:18.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:18.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:09:18.146 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:09:23.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:23.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:23.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043060 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f21cc036ee0 tx=0x7f21e40352d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:23.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043060 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f21cc036ee0 tx=0x7f21e40352d0).stop 2021-03-09T09:09:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043060 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:23.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:23.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044a70 con 0x7f21cc005e30 2021-03-09T09:09:23.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055700 con 0x7f21c4005da0 2021-03-09T09:09:23.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035180 con 0x7f21c40033c0 2021-03-09T09:09:23.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:23.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:23.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:23.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:23.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:23.118 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6031 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:23.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:23.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:23.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:23.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:23.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21c40033c0 2021-03-09T09:09:23.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c40033c0 2021-03-09T09:09:23.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03ff10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:23.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0083520 con 0x7f21c40033c0 2021-03-09T09:09:23.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0083520 con 0x7f21c40033c0 2021-03-09T09:09:23.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0083160 con 0x7f21c40033c0 2021-03-09T09:09:23.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.120+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:23.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.120+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:23.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:23.120+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:28.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:28.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:28.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03ff10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:28.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03ff10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f21e007d6b0 tx=0x7f21e0003740).stop 2021-03-09T09:09:28.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03ff10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:28.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03ff10 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:28.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc042d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:28.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:28.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:28.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049e30 con 0x7f21cc005e30 2021-03-09T09:09:28.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0368f0 con 0x7f21c4005da0 2021-03-09T09:09:28.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046680 con 0x7f21ec0a7f50 2021-03-09T09:09:28.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:28.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:28.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:28.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:28.120 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6072 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:28.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc042d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:28.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:28.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:28.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:28.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:28.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095630 con 0x7f21ec0a7f50 2021-03-09T09:09:28.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21ec0a7f50 2021-03-09T09:09:28.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc042d20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f21e00984b0 tx=0x7f21e0028ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:28.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0083160 con 0x7f21ec0a7f50 2021-03-09T09:09:28.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21ec0a7f50 2021-03-09T09:09:28.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0083160 con 0x7f21ec0a7f50 2021-03-09T09:09:28.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.124+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:28.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:28.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:28.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:33.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc042d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:33.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc042d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:33.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc042d20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f21e00984b0 tx=0x7f21e0028ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:33.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc042d20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f21e00984b0 tx=0x7f21e0028ef0).stop 2021-03-09T09:09:33.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc042d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:33.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc042d20 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:33.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc036390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:33.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:33.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:33.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019ac0 con 0x7f21cc005e30 2021-03-09T09:09:33.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a90 con 0x7f21c4005da0 2021-03-09T09:09:33.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21c40033c0 2021-03-09T09:09:33.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:33.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:33.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:33.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:33.121 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6103 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:33.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc036390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:33.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:33.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:33.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:33.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:33.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c40033c0 2021-03-09T09:09:33.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095630 con 0x7f21c40033c0 2021-03-09T09:09:33.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc036390 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e007c650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:33.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075e70 con 0x7f21c40033c0 2021-03-09T09:09:33.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075e70 con 0x7f21c40033c0 2021-03-09T09:09:33.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0083160 con 0x7f21c40033c0 2021-03-09T09:09:33.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.124+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:33.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:33.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:33.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:38.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc036390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc036390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc036390 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e007c650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc036390 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e007c650).stop 2021-03-09T09:09:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc036390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc036390 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:09:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0046e0 con 0x7f21c4005da0 2021-03-09T09:09:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033b00 con 0x7f21ec0a7f50 2021-03-09T09:09:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:38.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:38.123 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6147 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:38.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:38.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040c40 con 0x7f21ec0a7f50 2021-03-09T09:09:38.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e330 con 0x7f21ec0a7f50 2021-03-09T09:09:38.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.120+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0429a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f21cc01da50 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:38.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07f6c0 con 0x7f21ec0a7f50 2021-03-09T09:09:38.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:09:38.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07f6c0 con 0x7f21ec0a7f50 2021-03-09T09:09:38.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.124+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:38.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:38.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:38.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:43.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:43.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:43.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0429a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f21cc01da50 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:43.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0429a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f21cc01da50 tx=0x7f21dc080000).stop 2021-03-09T09:09:43.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0429a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:43.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0429a0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:43.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:43.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:43.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:43.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:09:43.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0300a0 con 0x7f21c4005da0 2021-03-09T09:09:43.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0046e0 con 0x7f21c40033c0 2021-03-09T09:09:43.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:43.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:43.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:43.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:43.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:43.125 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6183 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:43.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac70 con 0x7f21c40033c0 2021-03-09T09:09:43.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00e400 con 0x7f21c40033c0 2021-03-09T09:09:43.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.115+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03f160 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f21e40045b0 tx=0x7f21e4004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:43.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040030 con 0x7f21c40033c0 2021-03-09T09:09:43.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009ba0 con 0x7f21c40033c0 2021-03-09T09:09:43.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c40033c0 2021-03-09T09:09:43.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.127+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:43.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.127+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:43.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:43.127+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:48.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:48.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:48.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03f160 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f21e40045b0 tx=0x7f21e4004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:48.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03f160 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f21e40045b0 tx=0x7f21e4004650).stop 2021-03-09T09:09:48.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03f160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:48.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03f160 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:48.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:48.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:48.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:48.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043b40 con 0x7f21cc005e30 2021-03-09T09:09:48.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928f0 con 0x7f21c4005da0 2021-03-09T09:09:48.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035180 con 0x7f21ec0a7f50 2021-03-09T09:09:48.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:48.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:48.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:48.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:48.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:48.129 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6218 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:48.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:48.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:48.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:48.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:48.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d5f0 con 0x7f21ec0a7f50 2021-03-09T09:09:48.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac70 con 0x7f21ec0a7f50 2021-03-09T09:09:48.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e400d9c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:48.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:09:48.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400d4b0 con 0x7f21ec0a7f50 2021-03-09T09:09:48.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a1ba0 con 0x7f21ec0a7f50 2021-03-09T09:09:48.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.123+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:48.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.127+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:48.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:48.127+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:48.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:09:48.196 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:09:51.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 msgr2=0x7f21cc0861f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:09:51.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 msgr2=0x7f21cc0861f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:51.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0861f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc006d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0861f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc006d70).stop 2021-03-09T09:09:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 msgr2=0x7f21cc0861f0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:09:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21cc083d30 2021-03-09T09:09:51.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T09:09:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 msgr2=0x7f21cc0861f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0861f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 0x7f21cc03ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:51.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 0x7f21cc03ef70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc006d70).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:53.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:53.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:53.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e400d9c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:53.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e400d9c0).stop 2021-03-09T09:09:53.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.119+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:09:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c200 con 0x7f21c4005da0 2021-03-09T09:09:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055700 con 0x7f21cc083d30 2021-03-09T09:09:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:53.150 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6256 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:53.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:53.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:53.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:53.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc033e00 con 0x7f21cc083d30 2021-03-09T09:09:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:09:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc083d30 2021-03-09T09:09:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21cc083d30 2021-03-09T09:09:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:09:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.131+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.131+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:53.131+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:09:58.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc03f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:09:58.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc03f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:58.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:58.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0003740).stop 2021-03-09T09:09:58.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc03f460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:58.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:58.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:58.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:58.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:58.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:09:58.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086660 con 0x7f21c4005da0 2021-03-09T09:09:58.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0388a0 con 0x7f21ec0a7f50 2021-03-09T09:09:58.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:09:58.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:58.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:09:58.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:58.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:58.130 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6286 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:09:58.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:58.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:58.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:58.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:58.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0137d0 con 0x7f21ec0a7f50 2021-03-09T09:09:58.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040c40 con 0x7f21ec0a7f50 2021-03-09T09:09:58.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.127+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086170 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7f21dc0802a0 tx=0x7f21dc005000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:58.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21ec0a7f50 2021-03-09T09:09:58.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21ec0a7f50 2021-03-09T09:09:58.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21ec0a7f50 2021-03-09T09:09:58.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.131+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:09:58.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.135+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:09:58.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:09:58.135+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:03.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:03.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:03.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086170 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7f21dc0802a0 tx=0x7f21dc005000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:03.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086170 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7f21dc0802a0 tx=0x7f21dc005000).stop 2021-03-09T09:10:03.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:03.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086170 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:03.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:03.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:03.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:03.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0306c0 con 0x7f21cc005e30 2021-03-09T09:10:03.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a4a0 con 0x7f21c4005da0 2021-03-09T09:10:03.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02af40 con 0x7f21cc083d30 2021-03-09T09:10:03.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:03.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:03.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:03.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.123+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:03.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:03.137 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6319 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:03.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:03.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:03.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:03.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:03.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21cc083d30 2021-03-09T09:10:03.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02d5f0 con 0x7f21cc083d30 2021-03-09T09:10:03.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f21e40045b0 tx=0x7f21e4091fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:03.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089080 con 0x7f21cc083d30 2021-03-09T09:10:03.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027a60 con 0x7f21cc083d30 2021-03-09T09:10:03.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc083d30 2021-03-09T09:10:03.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.139+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:03.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.139+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:03.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:03.139+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:08.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc03f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:08.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc03f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:08.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f21e40045b0 tx=0x7f21e4091fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:08.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f21e40045b0 tx=0x7f21e4091fa0).stop 2021-03-09T09:10:08.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc03f460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:08.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc03f460 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:08.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc049a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:08.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:08.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:08.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089630 con 0x7f21cc005e30 2021-03-09T09:10:08.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095cd0 con 0x7f21c4005da0 2021-03-09T09:10:08.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023520 con 0x7f21ec0a7f50 2021-03-09T09:10:08.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:08.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:08.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:08.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc049a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:08.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:08.134 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:08.127+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6360 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:08.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:08.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:08.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:08.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:08.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0361d0 con 0x7f21ec0a7f50 2021-03-09T09:10:08.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc033e00 con 0x7f21ec0a7f50 2021-03-09T09:10:08.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc049a70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f21e003aa70 tx=0x7f21e007e4a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:08.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0028960 con 0x7f21ec0a7f50 2021-03-09T09:10:08.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00968a0 con 0x7f21ec0a7f50 2021-03-09T09:10:08.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:10:08.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.139+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:08.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.139+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:08.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:08.139+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:13.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc049a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:10:13.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc049a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:13.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc049a70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f21e003aa70 tx=0x7f21e007e4a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:13.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc049a70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f21e003aa70 tx=0x7f21e007e4a0).stop 2021-03-09T09:10:13.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc049a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:13.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc049a70 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:13.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:13.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:13.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:13.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928b0 con 0x7f21cc005e30 2021-03-09T09:10:13.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0236c0 con 0x7f21c4005da0 2021-03-09T09:10:13.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:10:13.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:13.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.127+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:13.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:13.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:13.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:13.136 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6401 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:13.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:13.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:13.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:13.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:13.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc033e00 con 0x7f21cc083d30 2021-03-09T09:10:13.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0361d0 con 0x7f21cc083d30 2021-03-09T09:10:13.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e00a6290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:13.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0080460 con 0x7f21cc083d30 2021-03-09T09:10:13.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0083e90 con 0x7f21cc083d30 2021-03-09T09:10:13.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc083d30 2021-03-09T09:10:13.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.139+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:13.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.139+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:13.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:13.139+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:18.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:10:18.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:18.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e00a6290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:18.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e00a6290).stop 2021-03-09T09:10:18.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:18.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:18.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:18.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:18.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:18.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:18.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:18.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f7a0 con 0x7f21cc005e30 2021-03-09T09:10:18.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009b00 con 0x7f21c4005da0 2021-03-09T09:10:18.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b0f0 con 0x7f21ec0a7f50 2021-03-09T09:10:18.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:18.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:18.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:18.144 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6434 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:18.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:18.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:18.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:18.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:18.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025500 con 0x7f21ec0a7f50 2021-03-09T09:10:18.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0137d0 con 0x7f21ec0a7f50 2021-03-09T09:10:18.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096200 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc006f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:18.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21ec0a7f50 2021-03-09T09:10:18.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21ec0a7f50 2021-03-09T09:10:18.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21ec0a7f50 2021-03-09T09:10:18.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.139+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:18.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:18.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:18.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:18.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:10:18.242 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:10:23.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc096200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:23.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc096200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:23.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096200 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc006f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:23.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096200 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc006f10).stop 2021-03-09T09:10:23.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc096200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:23.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096200 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:23.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:23.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:23.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:23.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928b0 con 0x7f21cc005e30 2021-03-09T09:10:23.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091c40 con 0x7f21c4005da0 2021-03-09T09:10:23.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b0f0 con 0x7f21cc083d30 2021-03-09T09:10:23.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:23.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:23.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:23.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:23.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.131+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:23.139 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6469 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:23.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:23.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:23.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:23.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:23.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095630 con 0x7f21cc083d30 2021-03-09T09:10:23.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21cc083d30 2021-03-09T09:10:23.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f21e403e280 tx=0x7f21e4091f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:23.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089090 con 0x7f21cc083d30 2021-03-09T09:10:23.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027a60 con 0x7f21cc083d30 2021-03-09T09:10:23.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc083d30 2021-03-09T09:10:23.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.143+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:23.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:23.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:23.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:28.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:10:28.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:28.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f21e403e280 tx=0x7f21e4091f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:28.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f21e403e280 tx=0x7f21e4091f30).stop 2021-03-09T09:10:28.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:28.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:28.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:28.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:28.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:28.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0937a0 con 0x7f21cc005e30 2021-03-09T09:10:28.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b160 con 0x7f21c4005da0 2021-03-09T09:10:28.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d940 con 0x7f21ec0a7f50 2021-03-09T09:10:28.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:28.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:28.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:28.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:28.141 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6506 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:28.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:28.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:28.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:28.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:28.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:28.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032830 con 0x7f21ec0a7f50 2021-03-09T09:10:28.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095630 con 0x7f21ec0a7f50 2021-03-09T09:10:28.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096960 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 rx=0x7f21cc03bee0 tx=0x7f21e40a5120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:28.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21ec0a7f50 2021-03-09T09:10:28.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21ec0a7f50 2021-03-09T09:10:28.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21ec0a7f50 2021-03-09T09:10:28.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.143+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:28.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.147+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:28.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:28.147+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:33.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc096960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:33.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc096960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:33.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096960 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 rx=0x7f21cc03bee0 tx=0x7f21e40a5120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:33.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096960 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 rx=0x7f21cc03bee0 tx=0x7f21e40a5120).stop 2021-03-09T09:10:33.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc096960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:33.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc096960 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:33.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:33.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:33.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:33.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0937a0 con 0x7f21cc005e30 2021-03-09T09:10:33.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053be0 con 0x7f21c4005da0 2021-03-09T09:10:33.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b070 con 0x7f21cc083d30 2021-03-09T09:10:33.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:33.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:33.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:33.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:33.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:33.143 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:33.135+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6540 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:33.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:33.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:33.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:33.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:33.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:10:33.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025500 con 0x7f21cc083d30 2021-03-09T09:10:33.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7f21dc006f80 tx=0x7f21dc0204f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:33.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053070 con 0x7f21cc083d30 2021-03-09T09:10:33.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc083d30 2021-03-09T09:10:33.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07e1d0 con 0x7f21cc083d30 2021-03-09T09:10:33.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.143+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:33.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.147+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:33.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:33.147+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:38.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:10:38.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:38.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7f21dc006f80 tx=0x7f21dc0204f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:38.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7f21dc006f80 tx=0x7f21dc0204f0).stop 2021-03-09T09:10:38.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:38.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:38.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:38.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:38.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:38.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0937a0 con 0x7f21cc005e30 2021-03-09T09:10:38.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053d80 con 0x7f21c4005da0 2021-03-09T09:10:38.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d630 con 0x7f21ec0a7f50 2021-03-09T09:10:38.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:38.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:38.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:38.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:38.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:38.145 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6577 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:38.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:38.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:38.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:38.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:38.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21ec0a7f50 2021-03-09T09:10:38.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc033e00 con 0x7f21ec0a7f50 2021-03-09T09:10:38.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.139+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0028fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:38.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21ec0a7f50 2021-03-09T09:10:38.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21ec0a7f50 2021-03-09T09:10:38.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0080460 con 0x7f21ec0a7f50 2021-03-09T09:10:38.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.147+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:38.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.147+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:38.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:38.147+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:43.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:10:43.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc08cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:43.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0028fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:43.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0028fa0).stop 2021-03-09T09:10:43.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc08cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc08cf70 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:43.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056e20 con 0x7f21cc005e30 2021-03-09T09:10:43.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0540e0 con 0x7f21c4005da0 2021-03-09T09:10:43.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019120 con 0x7f21cc083d30 2021-03-09T09:10:43.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:43.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:43.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:43.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:43.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:43.147 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:43.138+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6611 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:43.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:43.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:43.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:43.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:43.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0137d0 con 0x7f21cc083d30 2021-03-09T09:10:43.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032830 con 0x7f21cc083d30 2021-03-09T09:10:43.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f21cc034d80 tx=0x7f21e4027a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:43.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040020 con 0x7f21cc083d30 2021-03-09T09:10:43.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40807b0 con 0x7f21cc083d30 2021-03-09T09:10:43.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:10:43.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.146+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:43.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.146+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:43.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:43.146+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:48.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:10:48.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:48.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f21cc034d80 tx=0x7f21e4027a60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:48.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f21cc034d80 tx=0x7f21e4027a60).stop 2021-03-09T09:10:48.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:48.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:48.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:48.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:48.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:48.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dba0 con 0x7f21cc005e30 2021-03-09T09:10:48.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e5a0 con 0x7f21c4005da0 2021-03-09T09:10:48.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a940 con 0x7f21ec0a7f50 2021-03-09T09:10:48.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:48.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:48.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:48.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:48.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:48.148 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6647 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:48.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:48.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:48.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:48.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:48.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e300 con 0x7f21ec0a7f50 2021-03-09T09:10:48.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0137d0 con 0x7f21ec0a7f50 2021-03-09T09:10:48.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bc60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f21cc04f6f0 tx=0x7f21e4087d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:48.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21ec0a7f50 2021-03-09T09:10:48.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4052bc0 con 0x7f21ec0a7f50 2021-03-09T09:10:48.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407d2c0 con 0x7f21ec0a7f50 2021-03-09T09:10:48.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.154+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:48.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:48.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:48.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:48.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:10:48.285 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:10:53.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:10:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bc60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f21cc04f6f0 tx=0x7f21e4087d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bc60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f21cc04f6f0 tx=0x7f21e4087d10).stop 2021-03-09T09:10:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bc60 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b140 con 0x7f21cc005e30 2021-03-09T09:10:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac80 con 0x7f21c4005da0 2021-03-09T09:10:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ea0 con 0x7f21cc083d30 2021-03-09T09:10:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:53.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:53.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:53.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:53.151 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:53.142+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6685 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc045f50 con 0x7f21cc083d30 2021-03-09T09:10:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21cc083d30 2021-03-09T09:10:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f21e0089010 tx=0x7f21e0097eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:53.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0077900 con 0x7f21cc083d30 2021-03-09T09:10:53.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc083d30 2021-03-09T09:10:53.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:10:53.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.154+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:53.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:53.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:53.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:10:58.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:10:58.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:58.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f21e0089010 tx=0x7f21e0097eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:58.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f21e0089010 tx=0x7f21e0097eb0).stop 2021-03-09T09:10:58.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:58.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:58.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:58.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:58.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:58.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ea0 con 0x7f21cc005e30 2021-03-09T09:10:58.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019120 con 0x7f21c4005da0 2021-03-09T09:10:58.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0578e0 con 0x7f21ec0a7f50 2021-03-09T09:10:58.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:10:58.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:58.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:10:58.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:58.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:58.152 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6721 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:10:58.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:58.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0230c0 con 0x7f21ec0a7f50 2021-03-09T09:10:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:10:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bbf0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7f21dc0277b0 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053070 con 0x7f21ec0a7f50 2021-03-09T09:10:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21ec0a7f50 2021-03-09T09:10:58.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:10:58.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.154+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:10:58.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:10:58.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:10:58.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:03.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:03.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:03.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bbf0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7f21dc0277b0 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:03.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bbf0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7f21dc0277b0 tx=0x7f21dc07ff90).stop 2021-03-09T09:11:03.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:03.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bbf0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:03.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:03.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:03.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:03.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03cca0 con 0x7f21cc005e30 2021-03-09T09:11:03.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08ea10 con 0x7f21c4005da0 2021-03-09T09:11:03.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b00 con 0x7f21cc083d30 2021-03-09T09:11:03.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:03.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:03.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:03.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:03.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:03.154 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:03.146+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6757 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:03.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:03.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:03.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:03.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:03.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032070 con 0x7f21cc083d30 2021-03-09T09:11:03.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc045f50 con 0x7f21cc083d30 2021-03-09T09:11:03.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e009af90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:03.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc083d30 2021-03-09T09:11:03.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc083d30 2021-03-09T09:11:03.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:11:03.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.158+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:03.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:03.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:03.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:08.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:08.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:08.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e009af90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:08.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e009af90).stop 2021-03-09T09:11:08.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:08.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:08.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:08.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:08.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:08.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e1b0 con 0x7f21cc005e30 2021-03-09T09:11:08.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015830 con 0x7f21c4005da0 2021-03-09T09:11:08.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f650 con 0x7f21ec0a7f50 2021-03-09T09:11:08.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:08.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:08.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:08.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:08.158 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6793 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:08.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:08.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:08.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:08.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:08.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:08.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032830 con 0x7f21ec0a7f50 2021-03-09T09:11:08.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032070 con 0x7f21ec0a7f50 2021-03-09T09:11:08.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bb30 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:08.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0077a40 con 0x7f21ec0a7f50 2021-03-09T09:11:08.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0077d80 con 0x7f21ec0a7f50 2021-03-09T09:11:08.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0077a40 con 0x7f21ec0a7f50 2021-03-09T09:11:08.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.158+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:08.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:08.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:08.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:13.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:13.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:13.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bb30 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:13.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bb30 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c5a0).stop 2021-03-09T09:11:13.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00bb30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:13.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00bb30 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:13.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:13.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:13.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:13.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015830 con 0x7f21cc005e30 2021-03-09T09:11:13.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ea0 con 0x7f21c4005da0 2021-03-09T09:11:13.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0893f0 con 0x7f21cc083d30 2021-03-09T09:11:13.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:13.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:13.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:13.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:13.564 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6828 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:13.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:13.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:13.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:13.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:13.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:13.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dfa0 con 0x7f21cc083d30 2021-03-09T09:11:13.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032830 con 0x7f21cc083d30 2021-03-09T09:11:13.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f21e001e770 tx=0x7f21e0080a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:13.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:11:13.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0077a40 con 0x7f21cc083d30 2021-03-09T09:11:13.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0077d80 con 0x7f21cc083d30 2021-03-09T09:11:13.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.158+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:13.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:13.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:13.158+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:18.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:18.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:18.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f21e001e770 tx=0x7f21e0080a80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:18.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f21e001e770 tx=0x7f21e0080a80).stop 2021-03-09T09:11:18.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:18.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:18.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:18.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:18.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:18.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0564b0 con 0x7f21cc005e30 2021-03-09T09:11:18.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b7f0 con 0x7f21c4005da0 2021-03-09T09:11:18.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032830 con 0x7f21ec0a7f50 2021-03-09T09:11:18.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:18.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:18.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:18.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:18.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:18.159 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6862 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:18.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:18.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:18.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:18.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:18.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0317a0 con 0x7f21ec0a7f50 2021-03-09T09:11:18.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21ec0a7f50 2021-03-09T09:11:18.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ba90 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f21e007ad10 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:18.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0040 con 0x7f21ec0a7f50 2021-03-09T09:11:18.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0059000 con 0x7f21ec0a7f50 2021-03-09T09:11:18.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0077d80 con 0x7f21ec0a7f50 2021-03-09T09:11:18.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.166+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:18.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:18.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:18.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:18.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:11:18.344 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:11:23.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:23.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:23.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ba90 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f21e007ad10 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:23.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ba90 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f21e007ad10 tx=0x7f21e000b3e0).stop 2021-03-09T09:11:23.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00ba90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:23.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00ba90 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:23.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:23.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:23.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:23.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037cf0 con 0x7f21cc005e30 2021-03-09T09:11:23.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041460 con 0x7f21c4005da0 2021-03-09T09:11:23.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e9c0 con 0x7f21cc083d30 2021-03-09T09:11:23.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:23.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:23.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:23.162 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6899 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:23.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:23.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:23.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:23.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:23.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:23.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:23.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21cc083d30 2021-03-09T09:11:23.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0230c0 con 0x7f21cc083d30 2021-03-09T09:11:23.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f21cc093660 tx=0x7f21dc003870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:23.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0158b0 con 0x7f21cc083d30 2021-03-09T09:11:23.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0278a0 con 0x7f21cc083d30 2021-03-09T09:11:23.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc083d30 2021-03-09T09:11:23.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.162+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:23.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:23.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:23.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:28.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:28.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:28.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f21cc093660 tx=0x7f21dc003870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:28.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f21cc093660 tx=0x7f21dc003870).stop 2021-03-09T09:11:28.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:28.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:28.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:28.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:28.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:28.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc090260 con 0x7f21cc005e30 2021-03-09T09:11:28.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046620 con 0x7f21c4005da0 2021-03-09T09:11:28.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00fb60 con 0x7f21ec0a7f50 2021-03-09T09:11:28.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:28.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:28.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:28.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.154+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:28.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:28.164 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6930 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:28.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:28.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:28.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:28.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:28.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036550 con 0x7f21ec0a7f50 2021-03-09T09:11:28.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21ec0a7f50 2021-03-09T09:11:28.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c40 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0916a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:28.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc091070 con 0x7f21ec0a7f50 2021-03-09T09:11:28.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0911d0 con 0x7f21ec0a7f50 2021-03-09T09:11:28.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc091070 con 0x7f21ec0a7f50 2021-03-09T09:11:28.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.166+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:28.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:28.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:28.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:33.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc040c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:33.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc040c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:33.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c40 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0916a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:33.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c40 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0916a0).stop 2021-03-09T09:11:33.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc040c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:33.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c40 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:33.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:33.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:33.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:33.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00acd0 con 0x7f21cc005e30 2021-03-09T09:11:33.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21c4005da0 2021-03-09T09:11:33.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ec30 con 0x7f21cc083d30 2021-03-09T09:11:33.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:33.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:33.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:33.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:33.166 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6973 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:33.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:33.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:33.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:33.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:33.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:33.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03aba0 con 0x7f21cc083d30 2021-03-09T09:11:33.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.158+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e300 con 0x7f21cc083d30 2021-03-09T09:11:33.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f21e407d260 tx=0x7f21e4091260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:33.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089020 con 0x7f21cc083d30 2021-03-09T09:11:33.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc083d30 2021-03-09T09:11:33.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031230 con 0x7f21cc083d30 2021-03-09T09:11:33.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.166+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:33.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.170+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:33.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:33.170+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:38.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:38.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:38.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f21e407d260 tx=0x7f21e4091260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:38.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f21e407d260 tx=0x7f21e4091260).stop 2021-03-09T09:11:38.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:38.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:38.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:38.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:38.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:38.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00acd0 con 0x7f21cc005e30 2021-03-09T09:11:38.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c330 con 0x7f21c4005da0 2021-03-09T09:11:38.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016c30 con 0x7f21ec0a7f50 2021-03-09T09:11:38.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:38.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:38.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:38.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:38.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:38.167 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7007 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:38.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:38.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:38.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:38.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:38.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0948a0 con 0x7f21ec0a7f50 2021-03-09T09:11:38.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036550 con 0x7f21ec0a7f50 2021-03-09T09:11:38.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c80 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f21dc0038e0 tx=0x7f21dc020560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:38.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07f160 con 0x7f21ec0a7f50 2021-03-09T09:11:38.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0918e0 con 0x7f21ec0a7f50 2021-03-09T09:11:38.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:11:38.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.166+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:38.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.170+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:38.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:38.170+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:43.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc040c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:43.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc040c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:43.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c80 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f21dc0038e0 tx=0x7f21dc020560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:43.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c80 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f21dc0038e0 tx=0x7f21dc020560).stop 2021-03-09T09:11:43.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc040c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:43.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc040c80 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:43.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:43.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:43.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:43.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:43.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21cc005e30 2021-03-09T09:11:43.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:43.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c330 con 0x7f21c4005da0 2021-03-09T09:11:43.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048a70 con 0x7f21cc083d30 2021-03-09T09:11:43.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:43.169 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7043 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:43.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:43.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:43.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:43.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:43.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:43.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:43.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:11:43.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03aba0 con 0x7f21cc083d30 2021-03-09T09:11:43.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f21cc015ad0 tx=0x7f21e4087ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:43.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:11:43.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091d60 con 0x7f21cc083d30 2021-03-09T09:11:43.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:11:43.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.169+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:43.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:43.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:43.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:48.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:48.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:48.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f21cc015ad0 tx=0x7f21e4087ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:48.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f21cc015ad0 tx=0x7f21e4087ef0).stop 2021-03-09T09:11:48.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:48.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:48.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:48.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:48.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:48.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21cc005e30 2021-03-09T09:11:48.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025750 con 0x7f21c4005da0 2021-03-09T09:11:48.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c330 con 0x7f21ec0a7f50 2021-03-09T09:11:48.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:48.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:48.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:48.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:48.170 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7082 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:48.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:48.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:48.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:48.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:48.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:48.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d110 con 0x7f21ec0a7f50 2021-03-09T09:11:48.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:11:48.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04d870 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f21cc00ea60 tx=0x7f21e409f310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:48.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:11:48.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e30 con 0x7f21ec0a7f50 2021-03-09T09:11:48.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40311a0 con 0x7f21ec0a7f50 2021-03-09T09:11:48.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.173+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:48.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:48.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:48.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:48.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:11:48.387 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:11:53.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:53.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:53.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04d870 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f21cc00ea60 tx=0x7f21e409f310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:53.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04d870 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f21cc00ea60 tx=0x7f21e409f310).stop 2021-03-09T09:11:53.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04d870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:53.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04d870 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:53.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:53.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:53.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:53.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00acd0 con 0x7f21cc005e30 2021-03-09T09:11:53.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c330 con 0x7f21c4005da0 2021-03-09T09:11:53.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a280 con 0x7f21cc083d30 2021-03-09T09:11:53.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:53.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:53.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:53.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:53.172 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7115 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:53.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:53.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:53.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:53.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:53.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:53.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21cc083d30 2021-03-09T09:11:53.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d110 con 0x7f21cc083d30 2021-03-09T09:11:53.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7f21cc04e8f0 tx=0x7f21e4028ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:53.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:11:53.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4052a50 con 0x7f21cc083d30 2021-03-09T09:11:53.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40311a0 con 0x7f21cc083d30 2021-03-09T09:11:53.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.173+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:53.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:53.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:53.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:11:58.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:11:58.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:58.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7f21cc04e8f0 tx=0x7f21e4028ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:58.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7f21cc04e8f0 tx=0x7f21e4028ee0).stop 2021-03-09T09:11:58.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:58.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:58.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:58.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:58.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:58.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:11:58.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091d70 con 0x7f21cc005e30 2021-03-09T09:11:58.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:58.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04df10 con 0x7f21c4005da0 2021-03-09T09:11:58.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e8d0 con 0x7f21ec0a7f50 2021-03-09T09:11:58.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:11:58.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:58.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:58.174 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:11:58.165+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7155 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:11:58.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:58.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:58.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:58.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:58.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:11:58.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21ec0a7f50 2021-03-09T09:11:58.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4036250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:58.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21ec0a7f50 2021-03-09T09:11:58.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5ec0 con 0x7f21ec0a7f50 2021-03-09T09:11:58.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40311a0 con 0x7f21ec0a7f50 2021-03-09T09:11:58.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.177+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:11:58.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.177+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:11:58.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:11:58.177+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:03.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:12:03.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:03.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4036250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:03.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4036250).stop 2021-03-09T09:12:03.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:03.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d80 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:03.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:03.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:03.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:03.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21cc005e30 2021-03-09T09:12:03.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016dd0 con 0x7f21c4005da0 2021-03-09T09:12:03.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024a90 con 0x7f21cc083d30 2021-03-09T09:12:03.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:03.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:03.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:03.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:03.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:03.177 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7187 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:03.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:03.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:03.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:03.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:03.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e140 con 0x7f21cc083d30 2021-03-09T09:12:03.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0948a0 con 0x7f21cc083d30 2021-03-09T09:12:03.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f21dc006e90 tx=0x7f21dc006ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:03.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0278a0 con 0x7f21cc083d30 2021-03-09T09:12:03.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0278a0 con 0x7f21cc083d30 2021-03-09T09:12:03.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc083d30 2021-03-09T09:12:03.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.181+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:03.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:03.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:03.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:08.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f21dc006e90 tx=0x7f21dc006ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f21dc006e90 tx=0x7f21dc006ec0).stop 2021-03-09T09:12:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032420 con 0x7f21cc005e30 2021-03-09T09:12:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ff50 con 0x7f21c4005da0 2021-03-09T09:12:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc018e40 con 0x7f21ec0a7f50 2021-03-09T09:12:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:08.178 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7222 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:08.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:08.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:08.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:08.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:08.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:08.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02cbd0 con 0x7f21ec0a7f50 2021-03-09T09:12:08.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e140 con 0x7f21ec0a7f50 2021-03-09T09:12:08.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043710 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f21cc037e00 tx=0x7f21dc027690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:08.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053070 con 0x7f21ec0a7f50 2021-03-09T09:12:08.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0703d0 con 0x7f21ec0a7f50 2021-03-09T09:12:08.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031420 con 0x7f21ec0a7f50 2021-03-09T09:12:08.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.181+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:08.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:08.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:08.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:13.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:12:13.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:13.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043710 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f21cc037e00 tx=0x7f21dc027690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:13.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043710 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f21cc037e00 tx=0x7f21dc027690).stop 2021-03-09T09:12:13.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc043710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:13.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc043710 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:13.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:13.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:13.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:13.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21cc005e30 2021-03-09T09:12:13.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016dd0 con 0x7f21c4005da0 2021-03-09T09:12:13.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0494a0 con 0x7f21cc083d30 2021-03-09T09:12:13.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:13.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:13.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:13.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:13.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:13.180 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7256 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:13.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:13.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:13.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc031930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc031930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d110 con 0x7f21cc083d30 2021-03-09T09:12:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0317a0 con 0x7f21cc083d30 2021-03-09T09:12:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e009df40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:13.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc083d30 2021-03-09T09:12:13.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc083d30 2021-03-09T09:12:13.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:12:13.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.185+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:13.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:13.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:13.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:18.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:18.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:18.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e009df40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:18.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e009df40).stop 2021-03-09T09:12:18.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:18.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:18.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:18.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:18.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:18.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04aa50 con 0x7f21cc005e30 2021-03-09T09:12:18.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc018fe0 con 0x7f21c4005da0 2021-03-09T09:12:18.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21ec0a7f50 2021-03-09T09:12:18.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:18.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:18.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:18.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:18.186 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7297 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:18.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:18.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0974d0 con 0x7f21ec0a7f50 2021-03-09T09:12:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02cbd0 con 0x7f21ec0a7f50 2021-03-09T09:12:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f21dc0851a0 tx=0x7f21dc015ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc070030 con 0x7f21ec0a7f50 2021-03-09T09:12:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085210 con 0x7f21ec0a7f50 2021-03-09T09:12:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0add30 con 0x7f21ec0a7f50 2021-03-09T09:12:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.181+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:18.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:18.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:12:18.444 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:12:23.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:23.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:23.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f21dc0851a0 tx=0x7f21dc015ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:23.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f21dc0851a0 tx=0x7f21dc015ad0).stop 2021-03-09T09:12:23.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04dbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:23.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:23.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:23.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:23.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:23.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:23.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049c80 con 0x7f21cc005e30 2021-03-09T09:12:23.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:23.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:23.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:23.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0550f0 con 0x7f21c4005da0 2021-03-09T09:12:23.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ad90 con 0x7f21cc083d30 2021-03-09T09:12:23.183 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7328 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:23.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:23.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:23.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:23.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:23.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:23.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031c90 con 0x7f21cc083d30 2021-03-09T09:12:23.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0974d0 con 0x7f21cc083d30 2021-03-09T09:12:23.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0adc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:23.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053030 con 0x7f21cc083d30 2021-03-09T09:12:23.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc053030 con 0x7f21cc083d30 2021-03-09T09:12:23.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f540 con 0x7f21cc083d30 2021-03-09T09:12:23.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.189+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:23.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:23.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:23.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:28.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:12:28.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:28.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0adc40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:28.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0adc40).stop 2021-03-09T09:12:28.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:28.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:28.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:28.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:28.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:28.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043c40 con 0x7f21cc005e30 2021-03-09T09:12:28.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e4f0 con 0x7f21c4005da0 2021-03-09T09:12:28.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049c80 con 0x7f21ec0a7f50 2021-03-09T09:12:28.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:28.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:28.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:28.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:28.185 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7366 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:28.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:28.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:28.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:28.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:28.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:28.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc004de0 con 0x7f21ec0a7f50 2021-03-09T09:12:28.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031c90 con 0x7f21ec0a7f50 2021-03-09T09:12:28.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f21dc0adaa0 tx=0x7f21dc090f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:28.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc090440 con 0x7f21ec0a7f50 2021-03-09T09:12:28.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc008830 con 0x7f21ec0a7f50 2021-03-09T09:12:28.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f540 con 0x7f21ec0a7f50 2021-03-09T09:12:28.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.185+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:28.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:28.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:28.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:33.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:33.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:33.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f21dc0adaa0 tx=0x7f21dc090f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:33.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f21dc0adaa0 tx=0x7f21dc090f00).stop 2021-03-09T09:12:33.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04dbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:33.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04dbd0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:33.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:33.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:33.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:33.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d8c0 con 0x7f21cc005e30 2021-03-09T09:12:33.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049830 con 0x7f21c4005da0 2021-03-09T09:12:33.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009ca0 con 0x7f21cc083d30 2021-03-09T09:12:33.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:33.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:33.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:33.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:33.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:33.187 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7398 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:33.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:33.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:33.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:33.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:33.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054330 con 0x7f21cc083d30 2021-03-09T09:12:33.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d110 con 0x7f21cc083d30 2021-03-09T09:12:33.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f21e00a2d20 tx=0x7f21e000b450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:33.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc083d30 2021-03-09T09:12:33.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc083d30 2021-03-09T09:12:33.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:12:33.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.188+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:33.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.188+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:33.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:33.188+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:38.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:12:38.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:38.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f21e00a2d20 tx=0x7f21e000b450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:38.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f21e00a2d20 tx=0x7f21e000b450).stop 2021-03-09T09:12:38.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:38.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:38.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:38.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:38.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:38.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a650 con 0x7f21cc005e30 2021-03-09T09:12:38.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036200 con 0x7f21c4005da0 2021-03-09T09:12:38.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00bbe0 con 0x7f21ec0a7f50 2021-03-09T09:12:38.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:38.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:38.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:38.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:38.188 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7443 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:38.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:38.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:38.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:38.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:38.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:38.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009190 con 0x7f21ec0a7f50 2021-03-09T09:12:38.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.180+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc004de0 con 0x7f21ec0a7f50 2021-03-09T09:12:38.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f990 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f21dc003510 tx=0x7f21dc085090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:38.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21ec0a7f50 2021-03-09T09:12:38.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc076880 con 0x7f21ec0a7f50 2021-03-09T09:12:38.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc060380 con 0x7f21ec0a7f50 2021-03-09T09:12:38.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.192+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:38.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:38.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:38.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:43.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:43.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:43.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f990 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f21dc003510 tx=0x7f21dc085090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:43.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f990 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f21dc003510 tx=0x7f21dc085090).stop 2021-03-09T09:12:43.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:43.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f990 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:43.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:43.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:43.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:43.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031e80 con 0x7f21cc005e30 2021-03-09T09:12:43.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0361d0 con 0x7f21c4005da0 2021-03-09T09:12:43.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f50 con 0x7f21cc083d30 2021-03-09T09:12:43.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:43.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:43.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:43.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:43.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:43.191 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7475 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:43.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:43.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:43.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:43.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:43.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040e20 con 0x7f21cc083d30 2021-03-09T09:12:43.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054330 con 0x7f21cc083d30 2021-03-09T09:12:43.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04e790 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f21e001fce0 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:43.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21cc083d30 2021-03-09T09:12:43.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21cc083d30 2021-03-09T09:12:43.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e001fd50 con 0x7f21cc083d30 2021-03-09T09:12:43.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.192+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:43.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:43.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:43.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:48.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:12:48.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:48.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04e790 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f21e001fce0 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:48.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04e790 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f21e001fce0 tx=0x7f21e007c5a0).stop 2021-03-09T09:12:48.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04e790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:48.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04e790 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:48.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:48.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:48.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:48.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21cc005e30 2021-03-09T09:12:48.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a610 con 0x7f21c4005da0 2021-03-09T09:12:48.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:48.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086600 con 0x7f21ec0a7f50 2021-03-09T09:12:48.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:48.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:48.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:48.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:48.193 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:48.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:48.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:48.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:48.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:48.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0467c0 con 0x7f21ec0a7f50 2021-03-09T09:12:48.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.184+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:12:48.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f790 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f21cc044e50 tx=0x7f21e408ed40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:48.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:12:48.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b4d40 con 0x7f21ec0a7f50 2021-03-09T09:12:48.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21ec0a7f50 2021-03-09T09:12:48.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.196+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:48.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.196+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:48.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:48.196+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:48.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:12:48.485 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:12:53.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:12:53.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:53.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f790 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f21cc044e50 tx=0x7f21e408ed40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:53.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f790 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f21cc044e50 tx=0x7f21e408ed40).stop 2021-03-09T09:12:53.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:53.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f790 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:53.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:53.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:53.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:53.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057770 con 0x7f21cc005e30 2021-03-09T09:12:53.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a610 con 0x7f21c4005da0 2021-03-09T09:12:53.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0550f0 con 0x7f21cc083d30 2021-03-09T09:12:53.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:53.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:53.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:53.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:53.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:53.195 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7543 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:53.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:53.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:53.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:53.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:53.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:12:53.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0467c0 con 0x7f21cc083d30 2021-03-09T09:12:53.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.192+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fc80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4005dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:53.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:12:53.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027c20 con 0x7f21cc083d30 2021-03-09T09:12:53.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4015c20 con 0x7f21cc083d30 2021-03-09T09:12:53.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.196+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:53.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.196+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:53.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:53.196+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:12:58.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:58.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:58.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fc80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4005dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:58.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fc80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4005dd0).stop 2021-03-09T09:12:58.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00fc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:58.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fc80 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:58.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:58.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:58.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:58.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b2e0 con 0x7f21cc005e30 2021-03-09T09:12:58.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f650 con 0x7f21c4005da0 2021-03-09T09:12:58.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0550f0 con 0x7f21ec0a7f50 2021-03-09T09:12:58.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:12:58.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:58.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:58.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:12:58.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:58.199 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7586 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:12:58.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:58.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:58.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:58.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:58.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040e20 con 0x7f21ec0a7f50 2021-03-09T09:12:58.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009190 con 0x7f21ec0a7f50 2021-03-09T09:12:58.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f3b0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f21dc076a40 tx=0x7f21dc028c70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:58.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc053070 con 0x7f21ec0a7f50 2021-03-09T09:12:58.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc076880 con 0x7f21ec0a7f50 2021-03-09T09:12:58.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:12:58.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.200+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:12:58.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.200+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:12:58.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:12:58.200+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:03.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:03.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:03.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f3b0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f21dc076a40 tx=0x7f21dc028c70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:03.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f3b0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f21dc076a40 tx=0x7f21dc028c70).stop 2021-03-09T09:13:03.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:03.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f3b0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:03.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:03.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:03.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:03.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0163d0 con 0x7f21cc005e30 2021-03-09T09:13:03.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004de0 con 0x7f21c4005da0 2021-03-09T09:13:03.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f8a0 con 0x7f21cc083d30 2021-03-09T09:13:03.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:03.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:03.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:03.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:03.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:03.200 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7620 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:03.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:03.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:03.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:03.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:03.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00bbe0 con 0x7f21cc083d30 2021-03-09T09:13:03.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.192+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc083d30 2021-03-09T09:13:03.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f21e0028bc0 tx=0x7f21e007cb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:03.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0028cc0 con 0x7f21cc083d30 2021-03-09T09:13:03.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21cc083d30 2021-03-09T09:13:03.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:13:03.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.204+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:03.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:03.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:03.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:08.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:08.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:08.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f21e0028bc0 tx=0x7f21e007cb20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:08.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f21e0028bc0 tx=0x7f21e007cb20).stop 2021-03-09T09:13:08.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:08.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:08.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:08.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:08.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:08.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0361d0 con 0x7f21cc005e30 2021-03-09T09:13:08.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030710 con 0x7f21c4005da0 2021-03-09T09:13:08.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0550f0 con 0x7f21ec0a7f50 2021-03-09T09:13:08.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:08.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:08.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:08.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:08.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:08.201 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7660 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:08.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:08.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:08.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:08.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:08.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0280f0 con 0x7f21ec0a7f50 2021-03-09T09:13:08.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:13:08.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f000 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4005ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:08.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:13:08.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409f0d0 con 0x7f21ec0a7f50 2021-03-09T09:13:08.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091d60 con 0x7f21ec0a7f50 2021-03-09T09:13:08.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.204+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:08.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:08.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:08.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:13.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:13.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:13.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f000 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4005ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:13.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f000 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4005ee0).stop 2021-03-09T09:13:13.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00f000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:13.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00f000 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:13.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:13.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:13.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:13.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0361d0 con 0x7f21cc005e30 2021-03-09T09:13:13.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21c4005da0 2021-03-09T09:13:13.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da30 con 0x7f21cc083d30 2021-03-09T09:13:13.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:13.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:13.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:13.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:13.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:13.204 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7692 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:13.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:13.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:13.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:13.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:13.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:13:13.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040e20 con 0x7f21cc083d30 2021-03-09T09:13:13.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc006ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:13.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08c1e0 con 0x7f21cc083d30 2021-03-09T09:13:13.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08c000 con 0x7f21cc083d30 2021-03-09T09:13:13.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc083d30 2021-03-09T09:13:13.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.204+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:13.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:13.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:13.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:18.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:18.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc006ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc006ef0).stop 2021-03-09T09:13:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:18.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:18.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:18.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:13:18.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21c4005da0 2021-03-09T09:13:18.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b2e0 con 0x7f21ec0a7f50 2021-03-09T09:13:18.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:18.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:18.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:18.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:18.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:18.206 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7731 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:18.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:18.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:18.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:18.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:18.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0980b0 con 0x7f21ec0a7f50 2021-03-09T09:13:18.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00bbe0 con 0x7f21ec0a7f50 2021-03-09T09:13:18.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013590 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7f21e00a7a20 tx=0x7f21e00090c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:18.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e009ae80 con 0x7f21ec0a7f50 2021-03-09T09:13:18.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0027460 con 0x7f21ec0a7f50 2021-03-09T09:13:18.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:13:18.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.204+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:18.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:18.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:18.204+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:18.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:13:18.542 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:13:23.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:23.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:23.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013590 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7f21e00a7a20 tx=0x7f21e00090c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013590 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7f21e00a7a20 tx=0x7f21e00090c0).stop 2021-03-09T09:13:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013590 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0361d0 con 0x7f21cc005e30 2021-03-09T09:13:23.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cba0 con 0x7f21c4005da0 2021-03-09T09:13:23.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032990 con 0x7f21cc083d30 2021-03-09T09:13:23.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:23.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:23.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:23.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:23.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:23.208 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7764 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:23.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:23.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:23.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:23.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:23.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016570 con 0x7f21cc083d30 2021-03-09T09:13:23.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc083d30 2021-03-09T09:13:23.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f21e007e750 tx=0x7f21e0028920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:23.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc083d30 2021-03-09T09:13:23.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006a80 con 0x7f21cc083d30 2021-03-09T09:13:23.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.208+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0028220 con 0x7f21cc083d30 2021-03-09T09:13:23.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.208+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:23.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.208+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:23.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:23.208+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:28.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:28.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:28.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f21e007e750 tx=0x7f21e0028920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:28.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f21e007e750 tx=0x7f21e0028920).stop 2021-03-09T09:13:28.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:28.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:28.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:28.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:13:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21c4005da0 2021-03-09T09:13:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0361d0 con 0x7f21ec0a7f50 2021-03-09T09:13:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:28.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:28.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:28.210 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7802 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:28.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040e20 con 0x7f21ec0a7f50 2021-03-09T09:13:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0280f0 con 0x7f21ec0a7f50 2021-03-09T09:13:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fcf0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f21e407d1a0 tx=0x7f21e4091190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:13:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:13:28.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:13:28.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.211+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:28.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.211+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:28.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:28.211+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:33.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:33.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:33.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fcf0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f21e407d1a0 tx=0x7f21e4091190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:33.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fcf0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f21e407d1a0 tx=0x7f21e4091190).stop 2021-03-09T09:13:33.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:33.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fcf0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:33.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:33.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:33.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:33.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ceb0 con 0x7f21cc005e30 2021-03-09T09:13:33.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011cc0 con 0x7f21c4005da0 2021-03-09T09:13:33.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc083d30 2021-03-09T09:13:33.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:33.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:33.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:33.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:33.212 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7836 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:33.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:33.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:33.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:33.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:33.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.203+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:33.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc090e00 con 0x7f21cc083d30 2021-03-09T09:13:33.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:13:33.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037aa0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc006c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:33.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21cc083d30 2021-03-09T09:13:33.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc083d30 2021-03-09T09:13:33.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc083d30 2021-03-09T09:13:33.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.215+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:33.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.215+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:33.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:33.215+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:38.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc037aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:38.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc037aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:38.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037aa0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc006c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:38.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037aa0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc006c00).stop 2021-03-09T09:13:38.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc037aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:38.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037aa0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:38.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:38.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:38.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:38.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011cc0 con 0x7f21cc005e30 2021-03-09T09:13:38.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b2b0 con 0x7f21c4005da0 2021-03-09T09:13:38.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044400 con 0x7f21ec0a7f50 2021-03-09T09:13:38.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:38.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:38.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:38.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:38.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:38.214 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:38.207+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7874 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:38.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:38.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:38.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:38.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:38.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:13:38.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc090e00 con 0x7f21ec0a7f50 2021-03-09T09:13:38.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f21cc08d9d0 tx=0x7f21dc02f7a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:38.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007b70 con 0x7f21ec0a7f50 2021-03-09T09:13:38.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007cd0 con 0x7f21ec0a7f50 2021-03-09T09:13:38.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21ec0a7f50 2021-03-09T09:13:38.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.215+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:38.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.215+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:38.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:38.215+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:43.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:43.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:43.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f21cc08d9d0 tx=0x7f21dc02f7a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:43.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f21cc08d9d0 tx=0x7f21dc02f7a0).stop 2021-03-09T09:13:43.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:43.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:43.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:43.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:43.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:43.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:13:43.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ceb0 con 0x7f21c4005da0 2021-03-09T09:13:43.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a80 con 0x7f21cc083d30 2021-03-09T09:13:43.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:43.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:43.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:43.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:43.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:43.219 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7908 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:43.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:43.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:43.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:43.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:43.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0280f0 con 0x7f21cc083d30 2021-03-09T09:13:43.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016570 con 0x7f21cc083d30 2021-03-09T09:13:43.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037c20 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 rx=0x7f21e0003500 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:43.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc083d30 2021-03-09T09:13:43.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc083d30 2021-03-09T09:13:43.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:13:43.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.219+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:43.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.219+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:43.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:43.219+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:48.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc037c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:48.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc037c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:48.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037c20 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 rx=0x7f21e0003500 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:48.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037c20 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 rx=0x7f21e0003500 tx=0x7f21e000b3e0).stop 2021-03-09T09:13:48.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc037c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:48.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc037c20 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:48.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:48.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:48.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:48.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f70 con 0x7f21cc005e30 2021-03-09T09:13:48.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010710 con 0x7f21c4005da0 2021-03-09T09:13:48.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a80 con 0x7f21ec0a7f50 2021-03-09T09:13:48.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:48.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:48.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:48.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:48.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:48.217 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:48.211+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7945 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:48.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:48.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:48.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:48.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:48.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc019ac0 con 0x7f21ec0a7f50 2021-03-09T09:13:48.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040e20 con 0x7f21ec0a7f50 2021-03-09T09:13:48.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f21cc01e360 tx=0x7f21e4059000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:48.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:13:48.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027c20 con 0x7f21ec0a7f50 2021-03-09T09:13:48.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400d4b0 con 0x7f21ec0a7f50 2021-03-09T09:13:48.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.219+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:48.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.219+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:48.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:48.219+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:48.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:13:48.618 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:13:53.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:13:53.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:53.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f21cc01e360 tx=0x7f21e4059000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:53.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f21cc01e360 tx=0x7f21e4059000).stop 2021-03-09T09:13:53.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:53.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:53.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:53.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:53.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:53.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b00 con 0x7f21cc005e30 2021-03-09T09:13:53.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011cc0 con 0x7f21c4005da0 2021-03-09T09:13:53.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c7b0 con 0x7f21cc083d30 2021-03-09T09:13:53.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:53.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:53.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:53.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.211+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:53.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:53.219 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7980 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:53.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:53.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:53.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:53.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:53.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21cc083d30 2021-03-09T09:13:53.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:13:53.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00e5b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f21dc07e1c0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:53.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07e230 con 0x7f21cc083d30 2021-03-09T09:13:53.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07e000 con 0x7f21cc083d30 2021-03-09T09:13:53.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.227+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc083d30 2021-03-09T09:13:53.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.227+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:53.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:53.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:53.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:13:58.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:58.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:58.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00e5b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f21dc07e1c0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:58.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00e5b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f21dc07e1c0 tx=0x7f21dc077de0).stop 2021-03-09T09:13:58.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:58.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00e5b0 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:58.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:58.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:58.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:58.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a80 con 0x7f21cc005e30 2021-03-09T09:13:58.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04bce0 con 0x7f21c4005da0 2021-03-09T09:13:58.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011cc0 con 0x7f21ec0a7f50 2021-03-09T09:13:58.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:13:58.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:58.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:13:58.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:58.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:58.222 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:13:58.215+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8014 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:13:58.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:58.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:58.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:58.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:58.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc090e00 con 0x7f21ec0a7f50 2021-03-09T09:13:58.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0280f0 con 0x7f21ec0a7f50 2021-03-09T09:13:58.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00037b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:58.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21ec0a7f50 2021-03-09T09:13:58.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21ec0a7f50 2021-03-09T09:13:58.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e000ae30 con 0x7f21ec0a7f50 2021-03-09T09:13:58.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.223+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:13:58.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:13:58.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:13:58.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:03.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.215+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:03.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.215+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:03.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.215+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00037b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:03.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.215+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00037b0).stop 2021-03-09T09:14:03.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:03.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:03.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:03.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:03.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:03.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025970 con 0x7f21cc005e30 2021-03-09T09:14:03.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a80 con 0x7f21c4005da0 2021-03-09T09:14:03.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ec30 con 0x7f21cc083d30 2021-03-09T09:14:03.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:03.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:03.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:03.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:03.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:03.224 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8051 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:03.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:03.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:03.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:03.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:03.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016710 con 0x7f21cc083d30 2021-03-09T09:14:03.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc090e00 con 0x7f21cc083d30 2021-03-09T09:14:03.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04fcd0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e00a3ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:03.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.227+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:14:03.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.227+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc083d30 2021-03-09T09:14:03.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.227+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a3000 con 0x7f21cc083d30 2021-03-09T09:14:03.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.227+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:03.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:03.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:03.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:08.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:08.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:08.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04fcd0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e00a3ac0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:08.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04fcd0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e00a3ac0).stop 2021-03-09T09:14:08.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04fcd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:08.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04fcd0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:08.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:08.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:08.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:08.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a80 con 0x7f21cc005e30 2021-03-09T09:14:08.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da30 con 0x7f21c4005da0 2021-03-09T09:14:08.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011cc0 con 0x7f21ec0a7f50 2021-03-09T09:14:08.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:08.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:08.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:08.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:08.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:08.226 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:08.219+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8089 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:08.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:08.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:08.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:08.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:08.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096db0 con 0x7f21ec0a7f50 2021-03-09T09:14:08.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:14:08.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc020270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:08.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc006de0 con 0x7f21ec0a7f50 2021-03-09T09:14:08.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:14:08.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc005db0 con 0x7f21ec0a7f50 2021-03-09T09:14:08.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.231+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:08.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.231+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:08.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:08.231+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:13.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:13.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:13.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc020270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:13.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc020270).stop 2021-03-09T09:14:13.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:13.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:13.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:13.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:13.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:13.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a80 con 0x7f21cc005e30 2021-03-09T09:14:13.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011450 con 0x7f21c4005da0 2021-03-09T09:14:13.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cba0 con 0x7f21cc083d30 2021-03-09T09:14:13.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:13.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:13.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:13.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:13.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:13.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:13.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:13.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:13.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:13.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f9c0 con 0x7f21cc083d30 2021-03-09T09:14:13.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096db0 con 0x7f21cc083d30 2021-03-09T09:14:13.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7f21cc04fc70 tx=0x7f21dc006b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:13.231 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8125 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:13.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21cc083d30 2021-03-09T09:14:13.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01f160 con 0x7f21cc083d30 2021-03-09T09:14:13.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.227+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21cc083d30 2021-03-09T09:14:13.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.227+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:13.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:13.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:13.227+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:18.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:14:18.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:18.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7f21cc04fc70 tx=0x7f21dc006b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:18.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7f21cc04fc70 tx=0x7f21dc006b90).stop 2021-03-09T09:14:18.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:18.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:18.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:18.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:18.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:18.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0356c0 con 0x7f21cc005e30 2021-03-09T09:14:18.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e920 con 0x7f21c4005da0 2021-03-09T09:14:18.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011cc0 con 0x7f21ec0a7f50 2021-03-09T09:14:18.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:18.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:18.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:18.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:18.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:18.231 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8163 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:18.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:18.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:18.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:18.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:18.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:14:18.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc019ac0 con 0x7f21ec0a7f50 2021-03-09T09:14:18.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.227+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e080 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f21cc0100f0 tx=0x7f21e4053f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:18.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040090 con 0x7f21ec0a7f50 2021-03-09T09:14:18.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009360 con 0x7f21ec0a7f50 2021-03-09T09:14:18.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027c20 con 0x7f21ec0a7f50 2021-03-09T09:14:18.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.239+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:18.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.239+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:18.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:18.239+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:18.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:14:18.662 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:14:23.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:14:23.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:23.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e080 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f21cc0100f0 tx=0x7f21e4053f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:23.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e080 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f21cc0100f0 tx=0x7f21e4053f40).stop 2021-03-09T09:14:23.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00e080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:23.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00e080 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:23.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:23.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:23.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:23.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0964a0 con 0x7f21cc005e30 2021-03-09T09:14:23.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f5a0 con 0x7f21c4005da0 2021-03-09T09:14:23.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21cc083d30 2021-03-09T09:14:23.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:23.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:23.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:23.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:23.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:23.233 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8197 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:23.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:23.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:23.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:23.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:23.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00e570 con 0x7f21cc083d30 2021-03-09T09:14:23.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:14:23.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.230+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f21e40b0eb0 tx=0x7f21e4088c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:23.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21cc083d30 2021-03-09T09:14:23.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4088cf0 con 0x7f21cc083d30 2021-03-09T09:14:23.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.234+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21cc083d30 2021-03-09T09:14:23.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.234+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:23.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.234+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:23.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:23.234+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:28.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:14:28.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:28.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f21e40b0eb0 tx=0x7f21e4088c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:28.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f21e40b0eb0 tx=0x7f21e4088c00).stop 2021-03-09T09:14:28.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:28.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:28.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:28.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:28.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:28.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024bc0 con 0x7f21cc005e30 2021-03-09T09:14:28.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4005da0 2021-03-09T09:14:28.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052a40 con 0x7f21ec0a7f50 2021-03-09T09:14:28.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:28.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:28.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:28.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:28.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.226+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:28.235 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8235 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:28.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:28.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:28.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:28.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:28.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02c3e0 con 0x7f21ec0a7f50 2021-03-09T09:14:28.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21ec0a7f50 2021-03-09T09:14:28.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00dd40 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007c620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:28.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21ec0a7f50 2021-03-09T09:14:28.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21ec0a7f50 2021-03-09T09:14:28.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:14:28.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.238+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:28.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.238+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:28.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:28.238+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:33.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:33.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:33.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00dd40 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007c620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:33.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00dd40 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f21e007d580 tx=0x7f21e007c620).stop 2021-03-09T09:14:33.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00dd40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:33.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00dd40 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:33.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:33.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:33.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:33.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d360 con 0x7f21cc005e30 2021-03-09T09:14:33.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4005da0 2021-03-09T09:14:33.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d150 con 0x7f21cc083d30 2021-03-09T09:14:33.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:33.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:33.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:33.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:33.237 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8269 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:33.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.230+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:33.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:33.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:33.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:33.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:33.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032940 con 0x7f21cc083d30 2021-03-09T09:14:33.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00e570 con 0x7f21cc083d30 2021-03-09T09:14:33.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.234+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02e6e0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e403e310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:33.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:14:33.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4088cf0 con 0x7f21cc083d30 2021-03-09T09:14:33.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b47d0 con 0x7f21cc083d30 2021-03-09T09:14:33.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.242+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:33.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.242+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:33.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:33.242+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:38.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:38.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:38.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02e6e0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e403e310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:38.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02e6e0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e403e310).stop 2021-03-09T09:14:38.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02e6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:38.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02e6e0 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:38.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:38.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:38.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:38.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0461a0 con 0x7f21cc005e30 2021-03-09T09:14:38.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00afa0 con 0x7f21c4005da0 2021-03-09T09:14:38.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c080 con 0x7f21ec0a7f50 2021-03-09T09:14:38.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:38.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.230+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:38.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:38.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:38.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:38.239 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:38.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:38.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:38.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:38.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:38.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:14:38.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02c3e0 con 0x7f21ec0a7f50 2021-03-09T09:14:38.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:38.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21ec0a7f50 2021-03-09T09:14:38.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007f790 con 0x7f21ec0a7f50 2021-03-09T09:14:38.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:14:38.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.242+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:38.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.242+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:38.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:38.242+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:43.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003510).stop 2021-03-09T09:14:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:43.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:43.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:43.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:43.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:43.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08de20 con 0x7f21cc005e30 2021-03-09T09:14:43.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023e00 con 0x7f21c4005da0 2021-03-09T09:14:43.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048fc0 con 0x7f21cc083d30 2021-03-09T09:14:43.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:43.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:43.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:43.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:43.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:43.242 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:43.234+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8347 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:43.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:43.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091d20 con 0x7f21cc083d30 2021-03-09T09:14:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f9c0 con 0x7f21cc083d30 2021-03-09T09:14:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f21cc041580 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:43.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc006e40 con 0x7f21cc083d30 2021-03-09T09:14:43.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0278a0 con 0x7f21cc083d30 2021-03-09T09:14:43.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc083d30 2021-03-09T09:14:43.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.246+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:43.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.246+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:43.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:43.246+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:48.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:14:48.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:48.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f21cc041580 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:48.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f21cc041580 tx=0x7f21dc077de0).stop 2021-03-09T09:14:48.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:48.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:48.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:48.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:48.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:48.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ee60 con 0x7f21cc005e30 2021-03-09T09:14:48.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f50 con 0x7f21c4005da0 2021-03-09T09:14:48.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0464c0 con 0x7f21ec0a7f50 2021-03-09T09:14:48.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:48.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:48.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:48.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:48.244 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8379 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0280f0 con 0x7f21ec0a7f50 2021-03-09T09:14:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091d20 con 0x7f21ec0a7f50 2021-03-09T09:14:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f21dc0070a0 tx=0x7f21dc02e890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05f1b0 con 0x7f21ec0a7f50 2021-03-09T09:14:48.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f1b0 con 0x7f21ec0a7f50 2021-03-09T09:14:48.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003b70 con 0x7f21ec0a7f50 2021-03-09T09:14:48.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.250+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:48.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.250+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:48.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:48.250+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:48.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:14:48.704 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:14:53.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:14:53.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:53.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f21dc0070a0 tx=0x7f21dc02e890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:53.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f21dc0070a0 tx=0x7f21dc02e890).stop 2021-03-09T09:14:53.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:53.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:53.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:53.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:53.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:53.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:53.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011ec0 con 0x7f21cc005e30 2021-03-09T09:14:53.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:53.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:53.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:53.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ee60 con 0x7f21c4005da0 2021-03-09T09:14:53.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044450 con 0x7f21cc083d30 2021-03-09T09:14:53.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:53.251 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:53.238+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8413 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:53.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:53.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025970 con 0x7f21cc083d30 2021-03-09T09:14:53.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0280f0 con 0x7f21cc083d30 2021-03-09T09:14:53.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f21dc027a70 tx=0x7f21dc02ea90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:53.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062e70 con 0x7f21cc083d30 2021-03-09T09:14:53.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062e70 con 0x7f21cc083d30 2021-03-09T09:14:53.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0500e0 con 0x7f21cc083d30 2021-03-09T09:14:53.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.250+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:53.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.250+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:53.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:53.250+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:14:58.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:14:58.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:58.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f21dc027a70 tx=0x7f21dc02ea90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:58.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f21dc027a70 tx=0x7f21dc02ea90).stop 2021-03-09T09:14:58.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:58.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:58.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:58.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:58.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:58.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030710 con 0x7f21cc005e30 2021-03-09T09:14:58.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025bf0 con 0x7f21c4005da0 2021-03-09T09:14:58.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046800 con 0x7f21ec0a7f50 2021-03-09T09:14:58.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:14:58.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:58.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:14:58.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:58.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:58.248 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8451 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:14:58.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:58.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:58.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:58.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:58.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:14:58.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21ec0a7f50 2021-03-09T09:14:58.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0115e0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f21e0077810 tx=0x7f21e00a6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:58.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21ec0a7f50 2021-03-09T09:14:58.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21ec0a7f50 2021-03-09T09:14:58.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:14:58.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.250+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:14:58.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.250+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:14:58.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:14:58.250+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:03.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:03.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:03.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0115e0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f21e0077810 tx=0x7f21e00a6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:03.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0115e0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f21e0077810 tx=0x7f21e00a6190).stop 2021-03-09T09:15:03.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0115e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:03.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0115e0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:03.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:03.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:03.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:03.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:03.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:03.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033b00 con 0x7f21cc005e30 2021-03-09T09:15:03.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052af0 con 0x7f21c4005da0 2021-03-09T09:15:03.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e570 con 0x7f21cc083d30 2021-03-09T09:15:03.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:03.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:03.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:03.250 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:03.242+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8485 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:03.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:03.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:03.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:03.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:03.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21cc083d30 2021-03-09T09:15:03.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025970 con 0x7f21cc083d30 2021-03-09T09:15:03.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f21dc026e40 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:03.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017d90 con 0x7f21cc083d30 2021-03-09T09:15:03.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc026ca0 con 0x7f21cc083d30 2021-03-09T09:15:03.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.254+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21cc083d30 2021-03-09T09:15:03.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.254+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:03.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.254+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:03.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:03.254+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:08.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:08.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:08.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f21dc026e40 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:08.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f21dc026e40 tx=0x7f21dc080000).stop 2021-03-09T09:15:08.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:08.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:08.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:08.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:08.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:08.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f90 con 0x7f21cc005e30 2021-03-09T09:15:08.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046b40 con 0x7f21c4005da0 2021-03-09T09:15:08.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e570 con 0x7f21ec0a7f50 2021-03-09T09:15:08.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:08.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:08.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:08.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:08.253 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8521 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:08.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:08.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:08.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:08.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:08.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:08.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc019840 con 0x7f21ec0a7f50 2021-03-09T09:15:08.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:15:08.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.250+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02bf00 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc01f9e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:08.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06f000 con 0x7f21ec0a7f50 2021-03-09T09:15:08.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:15:08.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05efa0 con 0x7f21ec0a7f50 2021-03-09T09:15:08.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.250+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:08.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.254+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:08.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:08.254+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:13.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:13.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:13.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02bf00 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc01f9e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:13.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02bf00 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc01f9e0).stop 2021-03-09T09:15:13.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02bf00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:13.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02bf00 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:13.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:13.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:13.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:13.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03abf0 con 0x7f21cc005e30 2021-03-09T09:15:13.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0282d0 con 0x7f21c4005da0 2021-03-09T09:15:13.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c5d0 con 0x7f21cc083d30 2021-03-09T09:15:13.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:13.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:13.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:13.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:13.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.246+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:13.254 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8556 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:13.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:13.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:13.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:13.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:13.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21cc083d30 2021-03-09T09:15:13.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032940 con 0x7f21cc083d30 2021-03-09T09:15:13.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.250+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f21cc057280 tx=0x7f21e407f250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:13.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.254+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4089070 con 0x7f21cc083d30 2021-03-09T09:15:13.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.254+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4018ef0 con 0x7f21cc083d30 2021-03-09T09:15:13.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.258+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc083d30 2021-03-09T09:15:13.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.258+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:13.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.258+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:13.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:13.258+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:18.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:15:18.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:18.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f21cc057280 tx=0x7f21e407f250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:18.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f21cc057280 tx=0x7f21e407f250).stop 2021-03-09T09:15:18.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:18.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050630 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:18.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:18.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:18.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:18.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f8c0 con 0x7f21cc005e30 2021-03-09T09:15:18.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043e90 con 0x7f21c4005da0 2021-03-09T09:15:18.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f390 con 0x7f21ec0a7f50 2021-03-09T09:15:18.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:18.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:18.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:18.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:18.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:18.258 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8584 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:18.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:18.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:18.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:18.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:18.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00bad0 con 0x7f21ec0a7f50 2021-03-09T09:15:18.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.250+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:15:18.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.254+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f21e00a0de0 tx=0x7f21e0075c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:18.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.254+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21ec0a7f50 2021-03-09T09:15:18.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.258+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21ec0a7f50 2021-03-09T09:15:18.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.262+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:15:18.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.262+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:18.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.262+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:18.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:18.262+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:18.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:15:18.751 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:15:23.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:23.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:23.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f21e00a0de0 tx=0x7f21e0075c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:23.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f21e00a0de0 tx=0x7f21e0075c20).stop 2021-03-09T09:15:23.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:23.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:23.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:23.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:23.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:23.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d9c0 con 0x7f21cc005e30 2021-03-09T09:15:23.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da30 con 0x7f21c4005da0 2021-03-09T09:15:23.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0302e0 con 0x7f21cc083d30 2021-03-09T09:15:23.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:23.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:23.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:23.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:23.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:23.259 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8626 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:23.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:23.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:23.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:23.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:23.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086610 con 0x7f21cc083d30 2021-03-09T09:15:23.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc098250 con 0x7f21cc083d30 2021-03-09T09:15:23.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bc90 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f21cc01c930 tx=0x7f21e4003a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:23.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400d600 con 0x7f21cc083d30 2021-03-09T09:15:23.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:15:23.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:15:23.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.265+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:23.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.265+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:23.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:23.265+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:28.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:28.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:28.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bc90 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f21cc01c930 tx=0x7f21e4003a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:28.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bc90 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f21cc01c930 tx=0x7f21e4003a10).stop 2021-03-09T09:15:28.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02bc90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:28.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bc90 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:28.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:28.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:28.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:28.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e5f0 con 0x7f21cc005e30 2021-03-09T09:15:28.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:28.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041010 con 0x7f21c4005da0 2021-03-09T09:15:28.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:28.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:28.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037760 con 0x7f21ec0a7f50 2021-03-09T09:15:28.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:28.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:28.261 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8663 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:28.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:28.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:28.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:28.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:28.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:15:28.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.253+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086610 con 0x7f21ec0a7f50 2021-03-09T09:15:28.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.257+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f21e4087050 tx=0x7f21e407dee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:28.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:15:28.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e30 con 0x7f21ec0a7f50 2021-03-09T09:15:28.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e403ddc0 con 0x7f21ec0a7f50 2021-03-09T09:15:28.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.261+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:28.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.261+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:28.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:28.261+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:33.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:15:33.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:33.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f21e4087050 tx=0x7f21e407dee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:33.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f21e4087050 tx=0x7f21e407dee0).stop 2021-03-09T09:15:33.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:33.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:33.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:33.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:33.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:33.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0200f0 con 0x7f21cc005e30 2021-03-09T09:15:33.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055490 con 0x7f21c4005da0 2021-03-09T09:15:33.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032900 con 0x7f21cc083d30 2021-03-09T09:15:33.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:33.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:33.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:33.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:33.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:33.263 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8700 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:33.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:33.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:33.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:33.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:33.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046ea0 con 0x7f21cc083d30 2021-03-09T09:15:33.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc083d30 2021-03-09T09:15:33.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e0075d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:33.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:15:33.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a3650 con 0x7f21cc083d30 2021-03-09T09:15:33.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:15:33.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.265+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:33.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.265+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:33.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:33.265+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:38.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:38.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:38.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e0075d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:38.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e0075d10).stop 2021-03-09T09:15:38.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc008c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:38.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:38.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:38.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:38.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:38.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086610 con 0x7f21cc005e30 2021-03-09T09:15:38.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f0c0 con 0x7f21c4005da0 2021-03-09T09:15:38.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f200 con 0x7f21ec0a7f50 2021-03-09T09:15:38.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:38.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:38.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:38.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:38.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:38.266 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:38.257+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8746 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:38.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:38.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:38.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:38.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:38.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:15:38.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046ea0 con 0x7f21ec0a7f50 2021-03-09T09:15:38.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f21e0007d90 tx=0x7f21e007c660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:38.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0030 con 0x7f21ec0a7f50 2021-03-09T09:15:38.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21ec0a7f50 2021-03-09T09:15:38.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21ec0a7f50 2021-03-09T09:15:38.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.269+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:38.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.269+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:38.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:38.269+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:43.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:43.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:43.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f21e0007d90 tx=0x7f21e007c660).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:43.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f21e0007d90 tx=0x7f21e007c660).stop 2021-03-09T09:15:43.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:43.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:43.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:43.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:43.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:43.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055490 con 0x7f21cc005e30 2021-03-09T09:15:43.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032900 con 0x7f21c4005da0 2021-03-09T09:15:43.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f340 con 0x7f21cc083d30 2021-03-09T09:15:43.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:43.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:43.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:43.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:43.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:43.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:43.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:43.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:43.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:43.269 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8771 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:43.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025970 con 0x7f21cc083d30 2021-03-09T09:15:43.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc083d30 2021-03-09T09:15:43.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7f21e0003860 tx=0x7f21e003d290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:43.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0040 con 0x7f21cc083d30 2021-03-09T09:15:43.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e001fdb0 con 0x7f21cc083d30 2021-03-09T09:15:43.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:15:43.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.269+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:43.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.269+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:43.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:43.269+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:48.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:15:48.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:48.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7f21e0003860 tx=0x7f21e003d290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:48.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7f21e0003860 tx=0x7f21e003d290).stop 2021-03-09T09:15:48.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc008c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:48.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc008c40 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:48.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:48.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:48.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:48.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049a20 con 0x7f21cc005e30 2021-03-09T09:15:48.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04df20 con 0x7f21c4005da0 2021-03-09T09:15:48.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a70 con 0x7f21ec0a7f50 2021-03-09T09:15:48.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:48.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:48.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:48.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:48.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:48.270 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:48.261+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:48.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:48.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:48.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:48.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:48.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:15:48.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:15:48.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.265+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f21e403e2e0 tx=0x7f21e4027e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:48.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040020 con 0x7f21ec0a7f50 2021-03-09T09:15:48.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407dd10 con 0x7f21ec0a7f50 2021-03-09T09:15:48.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21ec0a7f50 2021-03-09T09:15:48.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.273+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:48.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.273+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:48.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:48.273+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:48.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:15:48.791 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:15:53.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:15:53.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:53.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f21e403e2e0 tx=0x7f21e4027e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:53.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f21e403e2e0 tx=0x7f21e4027e30).stop 2021-03-09T09:15:53.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:53.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:53.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:53.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:53.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:53.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055490 con 0x7f21cc005e30 2021-03-09T09:15:53.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019c80 con 0x7f21c4005da0 2021-03-09T09:15:53.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008030 con 0x7f21cc083d30 2021-03-09T09:15:53.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:53.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:53.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:53.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:53.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:53.276 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8844 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:53.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:53.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:53.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:53.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:53.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:15:53.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.265+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc019840 con 0x7f21cc083d30 2021-03-09T09:15:53.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bbf0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f21dc04fc10 tx=0x7f21dc01f970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:53.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b0420 con 0x7f21cc083d30 2021-03-09T09:15:53.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05fd30 con 0x7f21cc083d30 2021-03-09T09:15:53.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05efa0 con 0x7f21cc083d30 2021-03-09T09:15:53.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.273+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:53.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.273+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:53.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:53.273+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:15:58.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:15:58.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:58.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bbf0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f21dc04fc10 tx=0x7f21dc01f970).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:58.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bbf0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f21dc04fc10 tx=0x7f21dc01f970).stop 2021-03-09T09:15:58.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02bbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02bbf0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f840 con 0x7f21cc005e30 2021-03-09T09:15:58.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f440 con 0x7f21c4005da0 2021-03-09T09:15:58.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21ec0a7f50 2021-03-09T09:15:58.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:15:58.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:58.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:15:58.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:58.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:58.274 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8885 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:15:58.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:58.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:58.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:58.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:58.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e220 con 0x7f21ec0a7f50 2021-03-09T09:15:58.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:15:58.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f21cc025bc0 tx=0x7f21e4028960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:58.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:15:58.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403e080 con 0x7f21ec0a7f50 2021-03-09T09:15:58.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.281+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21ec0a7f50 2021-03-09T09:15:58.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.281+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:15:58.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.281+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:15:58.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:15:58.281+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:03.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:03.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:03.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f21cc025bc0 tx=0x7f21e4028960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:03.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f21cc025bc0 tx=0x7f21e4028960).stop 2021-03-09T09:16:03.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:03.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:03.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21cc005e30 2021-03-09T09:16:03.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f840 con 0x7f21c4005da0 2021-03-09T09:16:03.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f440 con 0x7f21cc083d30 2021-03-09T09:16:03.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:03.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:03.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:03.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:03.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:03.279 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:03.269+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8914 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:03.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:03.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:03.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:03.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:03.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b020 con 0x7f21cc083d30 2021-03-09T09:16:03.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:16:03.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f21cc01e810 tx=0x7f21dc02eec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:03.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:16:03.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.281+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21cc083d30 2021-03-09T09:16:03.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.281+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21cc083d30 2021-03-09T09:16:03.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.281+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:03.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.281+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:03.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:03.281+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:08.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:08.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:08.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f21cc01e810 tx=0x7f21dc02eec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:08.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f21cc01e810 tx=0x7f21dc02eec0).stop 2021-03-09T09:16:08.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:08.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:08.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:08.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:08.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:08.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0905d0 con 0x7f21cc005e30 2021-03-09T09:16:08.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21c4005da0 2021-03-09T09:16:08.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008000 con 0x7f21ec0a7f50 2021-03-09T09:16:08.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:08.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:08.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:08.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.269+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:08.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:08.279 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8951 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a2a0 con 0x7f21ec0a7f50 2021-03-09T09:16:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b020 con 0x7f21ec0a7f50 2021-03-09T09:16:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f21cc08e6f0 tx=0x7f21dc076ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:08.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.277+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02f0f0 con 0x7f21ec0a7f50 2021-03-09T09:16:08.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.277+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc091240 con 0x7f21ec0a7f50 2021-03-09T09:16:08.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.281+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21ec0a7f50 2021-03-09T09:16:08.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.281+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:08.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.281+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:08.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:08.281+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:13.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:13.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:13.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f21cc08e6f0 tx=0x7f21dc076ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:13.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f21cc08e6f0 tx=0x7f21dc076ec0).stop 2021-03-09T09:16:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:13.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:13.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21cc005e30 2021-03-09T09:16:13.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fec0 con 0x7f21c4005da0 2021-03-09T09:16:13.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f420 con 0x7f21cc083d30 2021-03-09T09:16:13.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:13.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:13.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:13.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:13.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:13.282 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:13.273+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8999 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc098250 con 0x7f21cc083d30 2021-03-09T09:16:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a2a0 con 0x7f21cc083d30 2021-03-09T09:16:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc003be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:13.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06f000 con 0x7f21cc083d30 2021-03-09T09:16:13.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.277+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06f160 con 0x7f21cc083d30 2021-03-09T09:16:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.285+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06f000 con 0x7f21cc083d30 2021-03-09T09:16:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.285+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.285+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:13.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:13.285+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:18.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:18.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:18.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc003be0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:18.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc003be0).stop 2021-03-09T09:16:18.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:18.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:18.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:18.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:18.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:18.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f840 con 0x7f21cc005e30 2021-03-09T09:16:18.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03aee0 con 0x7f21c4005da0 2021-03-09T09:16:18.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:18.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f1e0 con 0x7f21ec0a7f50 2021-03-09T09:16:18.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:18.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:18.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:18.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:18.283 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9023 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:18.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:18.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:18.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:18.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:18.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:16:18.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e220 con 0x7f21ec0a7f50 2021-03-09T09:16:18.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7f21cc008170 tx=0x7f21e40039a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:18.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:16:18.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:16:18.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:16:18.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.284+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:18.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.284+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:18.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:18.284+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:18.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:16:18.838 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:16:23.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:23.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:23.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7f21cc008170 tx=0x7f21e40039a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:23.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7f21cc008170 tx=0x7f21e40039a0).stop 2021-03-09T09:16:23.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:23.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:23.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:23.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:23.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:23.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0905a0 con 0x7f21cc005e30 2021-03-09T09:16:23.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04def0 con 0x7f21c4005da0 2021-03-09T09:16:23.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f400 con 0x7f21cc083d30 2021-03-09T09:16:23.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:23.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:23.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:23.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:23.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:23.285 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9061 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:23.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:23.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:23.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:23.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:23.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054060 con 0x7f21cc083d30 2021-03-09T09:16:23.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025970 con 0x7f21cc083d30 2021-03-09T09:16:23.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04cb70 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f21e001fa70 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:23.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc083d30 2021-03-09T09:16:23.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc083d30 2021-03-09T09:16:23.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:16:23.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.288+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:23.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:23.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:23.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:28.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:28.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:28.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04cb70 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f21e001fa70 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:28.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04cb70 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f21e001fa70 tx=0x7f21e007c5a0).stop 2021-03-09T09:16:28.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:28.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04cb70 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:28.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:28.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:28.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:28.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04def0 con 0x7f21cc005e30 2021-03-09T09:16:28.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc019840 con 0x7f21c4005da0 2021-03-09T09:16:28.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e330 con 0x7f21ec0a7f50 2021-03-09T09:16:28.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:28.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:28.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:28.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:28.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:28.289 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9095 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:28.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:28.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:28.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:28.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:28.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d060 con 0x7f21ec0a7f50 2021-03-09T09:16:28.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc098250 con 0x7f21ec0a7f50 2021-03-09T09:16:28.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0b0420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:28.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06f5f0 con 0x7f21ec0a7f50 2021-03-09T09:16:28.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06f5f0 con 0x7f21ec0a7f50 2021-03-09T09:16:28.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21ec0a7f50 2021-03-09T09:16:28.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.288+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:28.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:28.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:28.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:33.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:16:33.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:33.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0b0420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:33.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0b0420).stop 2021-03-09T09:16:33.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:33.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:33.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:33.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:33.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:33.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21cc005e30 2021-03-09T09:16:33.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041010 con 0x7f21c4005da0 2021-03-09T09:16:33.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c5e0 con 0x7f21cc083d30 2021-03-09T09:16:33.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:33.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:33.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:33.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:33.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:33.290 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:33.280+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9132 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:33.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:33.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:33.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:33.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:33.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047040 con 0x7f21cc083d30 2021-03-09T09:16:33.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:16:33.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f21e409cd90 tx=0x7f21e4091d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:33.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040080 con 0x7f21cc083d30 2021-03-09T09:16:33.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027a60 con 0x7f21cc083d30 2021-03-09T09:16:33.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400d4b0 con 0x7f21cc083d30 2021-03-09T09:16:33.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.292+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:33.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:33.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:33.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:38.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:16:38.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:38.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f21e409cd90 tx=0x7f21e4091d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:38.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f21e409cd90 tx=0x7f21e4091d60).stop 2021-03-09T09:16:38.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:38.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:38.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:38.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:38.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:38.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024c30 con 0x7f21cc005e30 2021-03-09T09:16:38.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21c4005da0 2021-03-09T09:16:38.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a90 con 0x7f21ec0a7f50 2021-03-09T09:16:38.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:38.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:38.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:38.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:38.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:38.292 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:38.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:38.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:38.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:38.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:38.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03ae90 con 0x7f21ec0a7f50 2021-03-09T09:16:38.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054060 con 0x7f21ec0a7f50 2021-03-09T09:16:38.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e001f8b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:38.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21ec0a7f50 2021-03-09T09:16:38.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0059e40 con 0x7f21ec0a7f50 2021-03-09T09:16:38.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:16:38.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.292+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:38.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:38.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:38.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:43.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:43.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:43.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e001f8b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:43.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e001f8b0).stop 2021-03-09T09:16:43.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:43.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:43.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:43.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:43.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:43.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:43.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21cc005e30 2021-03-09T09:16:43.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:43.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089c50 con 0x7f21c4005da0 2021-03-09T09:16:43.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004dc0 con 0x7f21cc083d30 2021-03-09T09:16:43.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:43.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:43.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:43.295 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:43.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:43.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:43.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:43.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:43.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a2a0 con 0x7f21cc083d30 2021-03-09T09:16:43.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d060 con 0x7f21cc083d30 2021-03-09T09:16:43.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f21dc062850 tx=0x7f21dc0608b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:43.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:16:43.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:16:43.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21cc083d30 2021-03-09T09:16:43.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.296+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:43.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:43.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:43.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:48.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:16:48.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:48.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f21dc062850 tx=0x7f21dc0608b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:48.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f21dc062850 tx=0x7f21dc0608b0).stop 2021-03-09T09:16:48.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc023110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:48.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc023110 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:48.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:48.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:48.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:48.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0938d0 con 0x7f21cc005e30 2021-03-09T09:16:48.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b3c0 con 0x7f21c4005da0 2021-03-09T09:16:48.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0924e0 con 0x7f21ec0a7f50 2021-03-09T09:16:48.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:48.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:48.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:48.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:48.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:48.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:48.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:48.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:48.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:48.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:16:48.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084020 con 0x7f21ec0a7f50 2021-03-09T09:16:48.298 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9240 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:48.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f21e007c570 tx=0x7f21e00a61f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:48.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21ec0a7f50 2021-03-09T09:16:48.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0028c50 con 0x7f21ec0a7f50 2021-03-09T09:16:48.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:16:48.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.296+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:48.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:48.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:48.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:48.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:16:48.877 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:16:53.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:16:53.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:53.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f21e007c570 tx=0x7f21e00a61f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:53.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f21e007c570 tx=0x7f21e00a61f0).stop 2021-03-09T09:16:53.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:53.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:53.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:53.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:53.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:53.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089c50 con 0x7f21cc005e30 2021-03-09T09:16:53.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0569f0 con 0x7f21c4005da0 2021-03-09T09:16:53.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:53.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e5f0 con 0x7f21cc083d30 2021-03-09T09:16:53.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:53.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:53.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:53.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:53.301 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9279 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:53.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:53.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:53.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:53.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:53.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024990 con 0x7f21cc083d30 2021-03-09T09:16:53.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a2a0 con 0x7f21cc083d30 2021-03-09T09:16:53.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.292+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fcf0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc02f2b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:53.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:16:53.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21cc083d30 2021-03-09T09:16:53.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc060920 con 0x7f21cc083d30 2021-03-09T09:16:53.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.300+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:53.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:53.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:53.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:16:58.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:58.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:58.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fcf0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc02f2b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:58.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fcf0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc02f2b0).stop 2021-03-09T09:16:58.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00fcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:58.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00fcf0 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:58.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:58.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:58.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:58.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03abf0 con 0x7f21cc005e30 2021-03-09T09:16:58.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053cd0 con 0x7f21c4005da0 2021-03-09T09:16:58.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053e10 con 0x7f21ec0a7f50 2021-03-09T09:16:58.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:58.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:16:58.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:58.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:16:58.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:58.303 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9311 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:16:58.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:58.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:58.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:58.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:58.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053f50 con 0x7f21ec0a7f50 2021-03-09T09:16:58.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:16:58.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.296+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f21e000b420 tx=0x7f21e000b4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:58.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21ec0a7f50 2021-03-09T09:16:58.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0039430 con 0x7f21ec0a7f50 2021-03-09T09:16:58.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:16:58.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.304+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:16:58.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.304+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:16:58.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:16:58.304+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:03.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:17:03.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:03.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f21e000b420 tx=0x7f21e000b4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:03.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f21e000b420 tx=0x7f21e000b4d0).stop 2021-03-09T09:17:03.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:03.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:03.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:03.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052590 con 0x7f21cc005e30 2021-03-09T09:17:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4005da0 2021-03-09T09:17:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016c00 con 0x7f21cc083d30 2021-03-09T09:17:03.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:03.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:03.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:03.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:03.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:03.305 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9359 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:03.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:03.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:03.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:03.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:03.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04cae0 con 0x7f21cc083d30 2021-03-09T09:17:03.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024990 con 0x7f21cc083d30 2021-03-09T09:17:03.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016710 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f21cc053c00 tx=0x7f21dc02ee70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:03.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc083d30 2021-03-09T09:17:03.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc083d30 2021-03-09T09:17:03.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc060920 con 0x7f21cc083d30 2021-03-09T09:17:03.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.304+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:03.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:03.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:03.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:08.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:08.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:08.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016710 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f21cc053c00 tx=0x7f21dc02ee70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:08.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016710 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f21cc053c00 tx=0x7f21dc02ee70).stop 2021-03-09T09:17:08.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:08.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016710 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:08.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:08.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:08.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:08.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:17:08.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024990 con 0x7f21c4005da0 2021-03-09T09:17:08.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048710 con 0x7f21ec0a7f50 2021-03-09T09:17:08.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:08.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:08.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:08.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:08.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:08.307 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9389 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:08.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:08.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:08.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:08.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:08.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc048880 con 0x7f21ec0a7f50 2021-03-09T09:17:08.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04cae0 con 0x7f21ec0a7f50 2021-03-09T09:17:08.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc06d400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:08.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21ec0a7f50 2021-03-09T09:17:08.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:17:08.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06d000 con 0x7f21ec0a7f50 2021-03-09T09:17:08.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.304+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:08.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:08.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:08.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:13.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:13.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:13.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc06d400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:13.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc06d400).stop 2021-03-09T09:17:13.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:13.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:13.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:13.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:13.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:13.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d160 con 0x7f21cc005e30 2021-03-09T09:17:13.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048710 con 0x7f21c4005da0 2021-03-09T09:17:13.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:17:13.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:13.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:13.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:13.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:13.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:13.309 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9429 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:13.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:13.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:13.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:13.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:13.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024990 con 0x7f21cc083d30 2021-03-09T09:17:13.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc047040 con 0x7f21cc083d30 2021-03-09T09:17:13.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f21cc009430 tx=0x7f21e4053f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:13.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:17:13.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4020190 con 0x7f21cc083d30 2021-03-09T09:17:13.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.308+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21cc083d30 2021-03-09T09:17:13.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.308+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:13.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.312+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:13.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:13.312+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:18.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:18.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:18.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f21cc009430 tx=0x7f21e4053f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:18.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f21cc009430 tx=0x7f21e4053f40).stop 2021-03-09T09:17:18.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:18.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:18.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:18.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:18.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:18.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024d90 con 0x7f21cc005e30 2021-03-09T09:17:18.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d9c0 con 0x7f21c4005da0 2021-03-09T09:17:18.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d660 con 0x7f21ec0a7f50 2021-03-09T09:17:18.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:18.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:18.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:18.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:18.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:18.311 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9465 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:18.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:18.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:18.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:18.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:18.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:17:18.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024990 con 0x7f21ec0a7f50 2021-03-09T09:17:18.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f21e4017fb0 tx=0x7f21e409c8f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:18.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:17:18.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400d6c0 con 0x7f21ec0a7f50 2021-03-09T09:17:18.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091dd0 con 0x7f21ec0a7f50 2021-03-09T09:17:18.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.311+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:18.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.311+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:18.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:18.311+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:18.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:17:18.917 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:17:23.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:17:23.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f21e4017fb0 tx=0x7f21e409c8f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f21e4017fb0 tx=0x7f21e409c8f0).stop 2021-03-09T09:17:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024d90 con 0x7f21cc005e30 2021-03-09T09:17:23.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093fa0 con 0x7f21c4005da0 2021-03-09T09:17:23.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0501e0 con 0x7f21cc083d30 2021-03-09T09:17:23.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:23.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:23.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:23.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:23.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:23.314 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9492 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:23.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:23.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:23.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:23.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:23.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc010300 con 0x7f21cc083d30 2021-03-09T09:17:23.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc083d30 2021-03-09T09:17:23.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e003cc60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:23.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21cc083d30 2021-03-09T09:17:23.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc083d30 2021-03-09T09:17:23.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:17:23.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.315+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:23.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.315+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:23.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:23.315+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:28.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:17:28.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:28.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e003cc60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:28.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e003cc60).stop 2021-03-09T09:17:28.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:28.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:28.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:28.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:28.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:28.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ce60 con 0x7f21cc005e30 2021-03-09T09:17:28.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025580 con 0x7f21c4005da0 2021-03-09T09:17:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d9e0 con 0x7f21ec0a7f50 2021-03-09T09:17:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:28.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:28.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:28.317 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:28.311+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9527 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:28.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:28.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:28.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:28.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024990 con 0x7f21ec0a7f50 2021-03-09T09:17:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc010300 con 0x7f21ec0a7f50 2021-03-09T09:17:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f21e007ba60 tx=0x7f21e007b6e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21ec0a7f50 2021-03-09T09:17:28.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007be70 con 0x7f21ec0a7f50 2021-03-09T09:17:28.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003d010 con 0x7f21ec0a7f50 2021-03-09T09:17:28.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.323+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:28.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:28.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:28.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:33.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:17:33.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:33.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f21e007ba60 tx=0x7f21e007b6e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:33.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f21e007ba60 tx=0x7f21e007b6e0).stop 2021-03-09T09:17:33.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:33.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:33.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:33.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:33.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:33.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004dc0 con 0x7f21cc005e30 2021-03-09T09:17:33.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0975e0 con 0x7f21c4005da0 2021-03-09T09:17:33.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048710 con 0x7f21cc083d30 2021-03-09T09:17:33.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:33.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:33.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:33.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:33.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:33.318 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9564 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:33.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:33.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:33.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:33.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:33.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025580 con 0x7f21cc083d30 2021-03-09T09:17:33.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:17:33.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f21e407d290 tx=0x7f21e4090980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:33.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:17:33.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:17:33.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:17:33.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.323+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:33.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:33.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:33.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:38.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:17:38.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:38.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f21e407d290 tx=0x7f21e4090980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:38.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f21e407d290 tx=0x7f21e4090980).stop 2021-03-09T09:17:38.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:38.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:38.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:38.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:38.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:38.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055440 con 0x7f21cc005e30 2021-03-09T09:17:38.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4005da0 2021-03-09T09:17:38.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044260 con 0x7f21ec0a7f50 2021-03-09T09:17:38.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:38.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:38.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:38.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:38.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:38.324 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9598 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:38.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:38.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:38.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:38.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:38.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024990 con 0x7f21ec0a7f50 2021-03-09T09:17:38.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc048880 con 0x7f21ec0a7f50 2021-03-09T09:17:38.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f21dc007b80 tx=0x7f21dc069280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:38.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0079c0 con 0x7f21ec0a7f50 2021-03-09T09:17:38.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:17:38.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21ec0a7f50 2021-03-09T09:17:38.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.323+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:38.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:38.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:38.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:43.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:17:43.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:43.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f21dc007b80 tx=0x7f21dc069280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:43.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f21dc007b80 tx=0x7f21dc069280).stop 2021-03-09T09:17:43.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:43.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:43.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:43.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:43.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:43.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052de0 con 0x7f21cc005e30 2021-03-09T09:17:43.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d9c0 con 0x7f21c4005da0 2021-03-09T09:17:43.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:17:43.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:43.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:43.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:43.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:43.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:43.323 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9636 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:43.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:43.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:43.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:43.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:43.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03ce10 con 0x7f21cc083d30 2021-03-09T09:17:43.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024990 con 0x7f21cc083d30 2021-03-09T09:17:43.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f21cc03a6c0 tx=0x7f21dc01f0b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:43.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc083d30 2021-03-09T09:17:43.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21cc083d30 2021-03-09T09:17:43.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc083d30 2021-03-09T09:17:43.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.327+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:43.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:43.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:43.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:48.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:17:48.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:48.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f21cc03a6c0 tx=0x7f21dc01f0b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:48.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f21cc03a6c0 tx=0x7f21dc01f0b0).stop 2021-03-09T09:17:48.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:48.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:48.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:48.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:48.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:48.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:48.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:17:48.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:48.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024990 con 0x7f21c4005da0 2021-03-09T09:17:48.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:17:48.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:48.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:48.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:48.327 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:48.319+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9679 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:48.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:48.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:48.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:48.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:48.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08f690 con 0x7f21ec0a7f50 2021-03-09T09:17:48.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03ce10 con 0x7f21ec0a7f50 2021-03-09T09:17:48.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc006cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:48.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba080 con 0x7f21ec0a7f50 2021-03-09T09:17:48.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052810 con 0x7f21ec0a7f50 2021-03-09T09:17:48.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21ec0a7f50 2021-03-09T09:17:48.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.331+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:48.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:48.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:48.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:48.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:17:48.957 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:17:53.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:17:53.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:53.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc006cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:53.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc006cd0).stop 2021-03-09T09:17:53.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:53.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:53.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:53.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:53.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:53.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d9c0 con 0x7f21cc005e30 2021-03-09T09:17:53.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0302e0 con 0x7f21c4005da0 2021-03-09T09:17:53.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:17:53.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:53.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:53.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:53.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:53.328 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9708 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:53.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:53.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:53.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:53.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:53.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:53.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00e190 con 0x7f21cc083d30 2021-03-09T09:17:53.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08f690 con 0x7f21cc083d30 2021-03-09T09:17:53.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f21cc00b6c0 tx=0x7f21dc0b4090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:53.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc083d30 2021-03-09T09:17:53.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017d80 con 0x7f21cc083d30 2021-03-09T09:17:53.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc083d30 2021-03-09T09:17:53.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.335+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:53.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:53.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:53.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:17:58.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:17:58.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:58.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f21cc00b6c0 tx=0x7f21dc0b4090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:58.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f21cc00b6c0 tx=0x7f21dc0b4090).stop 2021-03-09T09:17:58.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0230b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:58.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0230b0 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:58.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:58.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:58.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc024630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:58.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092390 con 0x7f21cc005e30 2021-03-09T09:17:58.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4005da0 2021-03-09T09:17:58.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:17:58.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:17:58.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:58.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:17:58.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.323+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:58.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc024630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:58.331 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9742 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:17:58.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:58.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:58.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:58.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:58.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:17:58.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00e190 con 0x7f21ec0a7f50 2021-03-09T09:17:58.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc024630 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f21dc006d40 tx=0x7f21dc050f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:58.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21ec0a7f50 2021-03-09T09:17:58.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062680 con 0x7f21ec0a7f50 2021-03-09T09:17:58.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:17:58.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.335+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:17:58.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:17:58.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:17:58.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:03.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc024630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:03.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc024630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:03.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc024630 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f21dc006d40 tx=0x7f21dc050f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:03.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc024630 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f21dc006d40 tx=0x7f21dc050f10).stop 2021-03-09T09:18:03.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc024630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:03.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc024630 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:03.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:03.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:03.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:03.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d9c0 con 0x7f21cc005e30 2021-03-09T09:18:03.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c5e0 con 0x7f21c4005da0 2021-03-09T09:18:03.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0166f0 con 0x7f21cc083d30 2021-03-09T09:18:03.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:03.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:03.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:03.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:03.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:03.333 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:03.327+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9783 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:03.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:03.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:03.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:03.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:03.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:18:03.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086120 con 0x7f21cc083d30 2021-03-09T09:18:03.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f21cc0295f0 tx=0x7f21dc07e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:03.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc083d30 2021-03-09T09:18:03.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09a040 con 0x7f21cc083d30 2021-03-09T09:18:03.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21cc083d30 2021-03-09T09:18:03.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.339+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:03.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.343+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:03.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:03.343+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:08.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:18:08.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:08.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f21cc0295f0 tx=0x7f21dc07e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:08.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f21cc0295f0 tx=0x7f21dc07e000).stop 2021-03-09T09:18:08.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:08.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:08.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:08.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:08.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:08.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f310 con 0x7f21cc005e30 2021-03-09T09:18:08.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011e60 con 0x7f21c4005da0 2021-03-09T09:18:08.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f4a0 con 0x7f21ec0a7f50 2021-03-09T09:18:08.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:08.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:08.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:08.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:08.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:08.335 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9825 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:08.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:08.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:08.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:08.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:08.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:18:08.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:18:08.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:08.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21ec0a7f50 2021-03-09T09:18:08.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:18:08.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003ccd0 con 0x7f21ec0a7f50 2021-03-09T09:18:08.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.339+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:08.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.339+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:08.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:08.339+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:13.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:13.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:13.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:13.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0003820).stop 2021-03-09T09:18:13.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:13.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:13.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:13.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:13.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:13.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035490 con 0x7f21cc005e30 2021-03-09T09:18:13.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0166f0 con 0x7f21c4005da0 2021-03-09T09:18:13.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052b50 con 0x7f21cc083d30 2021-03-09T09:18:13.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:13.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:13.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:13.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:13.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:13.338 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:13.331+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:13.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:13.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:13.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:13.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:13.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ee40 con 0x7f21cc083d30 2021-03-09T09:18:13.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:18:13.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.335+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e007cbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:13.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:18:13.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003cda0 con 0x7f21cc083d30 2021-03-09T09:18:13.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0007c90 con 0x7f21cc083d30 2021-03-09T09:18:13.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.343+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:13.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.343+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:13.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:13.343+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:18.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:18.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:18.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e007cbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:18.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e007cbb0).stop 2021-03-09T09:18:18.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc016200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:18.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc016200 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:18.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:18.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:18.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:18.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011eb0 con 0x7f21cc005e30 2021-03-09T09:18:18.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030620 con 0x7f21c4005da0 2021-03-09T09:18:18.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0166f0 con 0x7f21ec0a7f50 2021-03-09T09:18:18.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:18.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:18.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:18.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:18.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:18.341 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:18.334+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9894 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:18.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:18.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:18.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:18.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:18.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ee40 con 0x7f21ec0a7f50 2021-03-09T09:18:18.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:18:18.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f21dc0562e0 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:18.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b8040 con 0x7f21ec0a7f50 2021-03-09T09:18:18.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018d10 con 0x7f21ec0a7f50 2021-03-09T09:18:18.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21ec0a7f50 2021-03-09T09:18:18.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.346+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:18.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:18.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:18.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:19.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:18:19.011 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:18:23.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:18:23.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:23.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f21dc0562e0 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:23.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f21dc0562e0 tx=0x7f21dc07ff90).stop 2021-03-09T09:18:23.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:23.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:23.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:23.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:23.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc024200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:23.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0307c0 con 0x7f21cc005e30 2021-03-09T09:18:23.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012050 con 0x7f21c4005da0 2021-03-09T09:18:23.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f870 con 0x7f21cc083d30 2021-03-09T09:18:23.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:23.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:23.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:23.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.334+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:23.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc024200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:23.347 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9924 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:23.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:23.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:23.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:23.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:23.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:18:23.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ee40 con 0x7f21cc083d30 2021-03-09T09:18:23.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc024200 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f21cc057340 tx=0x7f21dc0071d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:23.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba040 con 0x7f21cc083d30 2021-03-09T09:18:23.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc083d30 2021-03-09T09:18:23.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21cc083d30 2021-03-09T09:18:23.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.346+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:23.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:23.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:23.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:28.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc024200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:28.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc024200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:28.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc024200 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f21cc057340 tx=0x7f21dc0071d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:28.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc024200 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f21cc057340 tx=0x7f21dc0071d0).stop 2021-03-09T09:18:28.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc024200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:28.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc024200 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:28.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:28.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:28.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:28.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092390 con 0x7f21cc005e30 2021-03-09T09:18:28.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4005da0 2021-03-09T09:18:28.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ee40 con 0x7f21ec0a7f50 2021-03-09T09:18:28.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:28.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:28.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:28.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:28.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:28.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:28.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:28.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:28.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:28.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04e760 con 0x7f21ec0a7f50 2021-03-09T09:18:28.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:18:28.346 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:28.338+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9961 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:28.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f21dc076ff0 tx=0x7f21dc0070a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:28.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06d490 con 0x7f21ec0a7f50 2021-03-09T09:18:28.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc003fb0 con 0x7f21ec0a7f50 2021-03-09T09:18:28.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085bc0 con 0x7f21ec0a7f50 2021-03-09T09:18:28.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.350+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:28.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:28.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:28.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:33.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:18:33.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:33.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f21dc076ff0 tx=0x7f21dc0070a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:33.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f21dc076ff0 tx=0x7f21dc0070a0).stop 2021-03-09T09:18:33.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:33.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:33.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:33.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:33.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:33.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047040 con 0x7f21cc005e30 2021-03-09T09:18:33.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037f50 con 0x7f21c4005da0 2021-03-09T09:18:33.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0163f0 con 0x7f21cc083d30 2021-03-09T09:18:33.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:33.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:33.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:33.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:33.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:33.348 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9997 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:33.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:33.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:33.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:33.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:33.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03caf0 con 0x7f21cc083d30 2021-03-09T09:18:33.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025580 con 0x7f21cc083d30 2021-03-09T09:18:33.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4090f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:33.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040090 con 0x7f21cc083d30 2021-03-09T09:18:33.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:18:33.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:18:33.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.350+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:33.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:33.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:33.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:38.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:38.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:38.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4090f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:38.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e4090f10).stop 2021-03-09T09:18:38.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:38.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:38.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:38.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:38.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:38.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052540 con 0x7f21cc005e30 2021-03-09T09:18:38.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bd90 con 0x7f21c4005da0 2021-03-09T09:18:38.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037f50 con 0x7f21ec0a7f50 2021-03-09T09:18:38.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:38.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:38.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:38.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:38.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:38.350 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10039 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:38.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:38.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:38.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:38.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:38.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:18:38.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04e760 con 0x7f21ec0a7f50 2021-03-09T09:18:38.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7f21dc0818a0 tx=0x7f21dc07cc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:38.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081e10 con 0x7f21ec0a7f50 2021-03-09T09:18:38.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21ec0a7f50 2021-03-09T09:18:38.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc068d80 con 0x7f21ec0a7f50 2021-03-09T09:18:38.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.354+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:38.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:38.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:38.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:43.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:18:43.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:43.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7f21dc0818a0 tx=0x7f21dc07cc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:43.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7f21dc0818a0 tx=0x7f21dc07cc90).stop 2021-03-09T09:18:43.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:43.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:43.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:43.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:43.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:43.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bd90 con 0x7f21cc005e30 2021-03-09T09:18:43.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024a30 con 0x7f21c4005da0 2021-03-09T09:18:43.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024530 con 0x7f21cc083d30 2021-03-09T09:18:43.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:43.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:43.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:43.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:43.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:43.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:43.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:43.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:43.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:43.355 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10074 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:43.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092530 con 0x7f21cc083d30 2021-03-09T09:18:43.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03caf0 con 0x7f21cc083d30 2021-03-09T09:18:43.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7f21e4086c50 tx=0x7f21e4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:43.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040080 con 0x7f21cc083d30 2021-03-09T09:18:43.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40911f0 con 0x7f21cc083d30 2021-03-09T09:18:43.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:18:43.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.354+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:43.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:43.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:43.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:48.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7f21e4086c50 tx=0x7f21e4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7f21e4086c50 tx=0x7f21e4005110).stop 2021-03-09T09:18:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:48.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:48.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:48.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:48.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7b0 con 0x7f21cc005e30 2021-03-09T09:18:48.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044e30 con 0x7f21c4005da0 2021-03-09T09:18:48.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047040 con 0x7f21ec0a7f50 2021-03-09T09:18:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:48.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:48.355 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10100 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:48.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:48.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ee40 con 0x7f21ec0a7f50 2021-03-09T09:18:48.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:18:48.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00282e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:48.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:18:48.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:18:48.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21ec0a7f50 2021-03-09T09:18:48.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.362+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:48.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:48.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:48.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:49.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:18:49.083 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:18:53.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:53.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:53.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00282e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:53.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00282e0).stop 2021-03-09T09:18:53.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:53.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:53.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:53.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:53.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:53.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bd90 con 0x7f21cc005e30 2021-03-09T09:18:53.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037f50 con 0x7f21c4005da0 2021-03-09T09:18:53.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ce00 con 0x7f21cc083d30 2021-03-09T09:18:53.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:53.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:53.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:53.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:53.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:53.357 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:53.350+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10140 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:53.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:18:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:18:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7f21dc089810 tx=0x7f21dc078e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089e50 con 0x7f21cc083d30 2021-03-09T09:18:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089e50 con 0x7f21cc083d30 2021-03-09T09:18:53.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21cc083d30 2021-03-09T09:18:53.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.362+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:53.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:53.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:53.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:18:58.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:18:58.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:58.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7f21dc089810 tx=0x7f21dc078e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:58.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7f21dc089810 tx=0x7f21dc078e70).stop 2021-03-09T09:18:58.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:58.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:58.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:58.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:58.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:58.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:18:58.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:58.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bd90 con 0x7f21cc005e30 2021-03-09T09:18:58.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044fd0 con 0x7f21c4005da0 2021-03-09T09:18:58.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cee0 con 0x7f21ec0a7f50 2021-03-09T09:18:58.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:18:58.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:58.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:58.360 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10175 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:18:58.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:58.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:58.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:58.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:58.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:18:58.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:18:58.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f21dc07e7e0 tx=0x7f21dc07e690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:58.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21ec0a7f50 2021-03-09T09:18:58.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc003ac0 con 0x7f21ec0a7f50 2021-03-09T09:18:58.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21ec0a7f50 2021-03-09T09:18:58.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.362+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:18:58.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:18:58.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:18:58.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:03.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:03.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:03.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f21dc07e7e0 tx=0x7f21dc07e690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:03.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f21dc07e7e0 tx=0x7f21dc07e690).stop 2021-03-09T09:19:03.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:03.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:03.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:03.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:03.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:03.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cee0 con 0x7f21cc005e30 2021-03-09T09:19:03.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bd90 con 0x7f21c4005da0 2021-03-09T09:19:03.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044fd0 con 0x7f21cc083d30 2021-03-09T09:19:03.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:03.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:03.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:03.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:03.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:03.364 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10211 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:03.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:03.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:03.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:03.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:03.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:19:03.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ee40 con 0x7f21cc083d30 2021-03-09T09:19:03.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 rx=0x7f21e000a7b0 tx=0x7f21e0075c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:03.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:19:03.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000a5f0 con 0x7f21cc083d30 2021-03-09T09:19:03.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21cc083d30 2021-03-09T09:19:03.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.366+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:03.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.366+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:03.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:03.366+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:08.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:08.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:08.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 rx=0x7f21e000a7b0 tx=0x7f21e0075c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:08.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 rx=0x7f21e000a7b0 tx=0x7f21e0075c10).stop 2021-03-09T09:19:08.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:08.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:08.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:08.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:08.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:08.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0168b0 con 0x7f21cc005e30 2021-03-09T09:19:08.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044fd0 con 0x7f21c4005da0 2021-03-09T09:19:08.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21ec0a7f50 2021-03-09T09:19:08.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:08.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:08.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:08.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:08.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:08.366 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:08.358+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10247 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:08.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:08.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:08.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:08.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:08.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ee40 con 0x7f21ec0a7f50 2021-03-09T09:19:08.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21ec0a7f50 2021-03-09T09:19:08.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 rx=0x7f21e0090ac0 tx=0x7f21e0003990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:08.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d2a0 con 0x7f21ec0a7f50 2021-03-09T09:19:08.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0059730 con 0x7f21ec0a7f50 2021-03-09T09:19:08.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:19:08.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.370+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:08.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:08.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:08.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:13.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:19:13.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:13.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 rx=0x7f21e0090ac0 tx=0x7f21e0003990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:13.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 rx=0x7f21e0090ac0 tx=0x7f21e0003990).stop 2021-03-09T09:19:13.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:13.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:13.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:13.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:13.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:13.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21cc005e30 2021-03-09T09:19:13.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c9a0 con 0x7f21c4005da0 2021-03-09T09:19:13.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0570d0 con 0x7f21cc083d30 2021-03-09T09:19:13.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:13.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:13.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:13.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:13.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:13.368 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10286 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:13.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:13.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:13.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:13.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:13.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094aa0 con 0x7f21cc083d30 2021-03-09T09:19:13.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ee40 con 0x7f21cc083d30 2021-03-09T09:19:13.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f21e0097810 tx=0x7f21e00a3e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:13.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0097650 con 0x7f21cc083d30 2021-03-09T09:19:13.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0097470 con 0x7f21cc083d30 2021-03-09T09:19:13.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:19:13.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.370+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:13.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:13.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:13.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:18.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:18.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:18.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f21e0097810 tx=0x7f21e00a3e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:18.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f21e0097810 tx=0x7f21e00a3e30).stop 2021-03-09T09:19:18.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc050100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:18.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc050100 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:18.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:18.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:18.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:18.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c360 con 0x7f21cc005e30 2021-03-09T09:19:18.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21c4005da0 2021-03-09T09:19:18.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0908e0 con 0x7f21ec0a7f50 2021-03-09T09:19:18.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:18.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:18.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:18.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:18.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:18.370 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:18.361+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10318 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:18.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:18.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:18.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:18.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:18.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0168b0 con 0x7f21ec0a7f50 2021-03-09T09:19:18.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc092530 con 0x7f21ec0a7f50 2021-03-09T09:19:18.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7f21cc01b6d0 tx=0x7f21e4036410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:18.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027c20 con 0x7f21ec0a7f50 2021-03-09T09:19:18.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.365+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027da0 con 0x7f21ec0a7f50 2021-03-09T09:19:18.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400d4b0 con 0x7f21ec0a7f50 2021-03-09T09:19:18.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.373+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:18.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.373+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:18.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:18.373+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:19.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:19:19.157 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:19:23.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:23.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:23.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7f21cc01b6d0 tx=0x7f21e4036410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:23.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7f21cc01b6d0 tx=0x7f21e4036410).stop 2021-03-09T09:19:23.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc086120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:23.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc086120 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:23.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:23.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0895a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:23.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:23.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c360 con 0x7f21cc005e30 2021-03-09T09:19:23.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bd90 con 0x7f21c4005da0 2021-03-09T09:19:23.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047040 con 0x7f21cc083d30 2021-03-09T09:19:23.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:23.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:23.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0895a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:23.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:23.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:23.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:23.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:23.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:23.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:23.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047900 con 0x7f21cc083d30 2021-03-09T09:19:23.373 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10356 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:23.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0168b0 con 0x7f21cc083d30 2021-03-09T09:19:23.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.365+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0895a0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f21e4027bd0 tx=0x7f21e407dfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:23.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:19:23.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091ec0 con 0x7f21cc083d30 2021-03-09T09:19:23.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009300 con 0x7f21cc083d30 2021-03-09T09:19:23.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.373+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:23.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.373+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:23.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:23.373+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:28.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0895a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:28.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0895a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:28.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0895a0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f21e4027bd0 tx=0x7f21e407dfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:28.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0895a0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f21e4027bd0 tx=0x7f21e407dfb0).stop 2021-03-09T09:19:28.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0895a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:28.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0895a0 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:28.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:28.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:28.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:28.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21cc005e30 2021-03-09T09:19:28.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d650 con 0x7f21c4005da0 2021-03-09T09:19:28.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dba0 con 0x7f21ec0a7f50 2021-03-09T09:19:28.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:28.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:28.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:28.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:28.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:28.374 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10391 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:28.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:28.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:28.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:28.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:28.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:19:28.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:19:28.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00b360 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 rx=0x7f21e007cab0 tx=0x7f21e00037b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:28.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0007880 con 0x7f21ec0a7f50 2021-03-09T09:19:28.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21ec0a7f50 2021-03-09T09:19:28.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:19:28.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.373+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:28.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.377+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:28.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:28.377+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:33.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:19:33.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:33.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00b360 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 rx=0x7f21e007cab0 tx=0x7f21e00037b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:33.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00b360 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 rx=0x7f21e007cab0 tx=0x7f21e00037b0).stop 2021-03-09T09:19:33.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc00b360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:33.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc00b360 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:33.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:33.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:33.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:33.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21cc005e30 2021-03-09T09:19:33.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0241e0 con 0x7f21c4005da0 2021-03-09T09:19:33.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056650 con 0x7f21cc083d30 2021-03-09T09:19:33.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:33.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:33.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:33.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:33.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:33.376 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10428 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:33.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:33.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:33.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:33.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:33.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:19:33.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.369+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc047900 con 0x7f21cc083d30 2021-03-09T09:19:33.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089250 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7f21cc02c190 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:33.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:19:33.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4057670 con 0x7f21cc083d30 2021-03-09T09:19:33.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091180 con 0x7f21cc083d30 2021-03-09T09:19:33.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.377+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:33.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.381+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:33.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:33.381+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:38.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:19:38.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:38.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089250 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7f21cc02c190 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:38.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089250 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7f21cc02c190 tx=0x7f21e4053ed0).stop 2021-03-09T09:19:38.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:38.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089250 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:38.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:38.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:38.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:38.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21cc005e30 2021-03-09T09:19:38.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00fe50 con 0x7f21c4005da0 2021-03-09T09:19:38.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dba0 con 0x7f21ec0a7f50 2021-03-09T09:19:38.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:38.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:38.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:38.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:38.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:38.381 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10462 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:38.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:38.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:38.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:38.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:38.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011e90 con 0x7f21ec0a7f50 2021-03-09T09:19:38.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:19:38.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0500b0 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f21e407e040 tx=0x7f21e40900b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:38.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091180 con 0x7f21ec0a7f50 2021-03-09T09:19:38.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40912e0 con 0x7f21ec0a7f50 2021-03-09T09:19:38.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4057c70 con 0x7f21ec0a7f50 2021-03-09T09:19:38.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.381+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:38.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.385+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:38.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:38.385+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:43.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:19:43.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:43.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0500b0 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f21e407e040 tx=0x7f21e40900b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:43.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0500b0 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f21e407e040 tx=0x7f21e40900b0).stop 2021-03-09T09:19:43.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0500b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:43.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0500b0 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:43.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:43.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:43.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:43.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21cc005e30 2021-03-09T09:19:43.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00fe50 con 0x7f21c4005da0 2021-03-09T09:19:43.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e5f0 con 0x7f21cc083d30 2021-03-09T09:19:43.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:43.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:43.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:43.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:43.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:43.381 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:43.373+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10504 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:43.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:43.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:43.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:43.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:43.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052540 con 0x7f21cc083d30 2021-03-09T09:19:43.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011e90 con 0x7f21cc083d30 2021-03-09T09:19:43.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089700 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f21e408f0a0 tx=0x7f21e4091270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:43.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b14c0 con 0x7f21cc083d30 2021-03-09T09:19:43.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc083d30 2021-03-09T09:19:43.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:19:43.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.385+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:43.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.385+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:43.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:43.385+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:48.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:19:48.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:48.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089700 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f21e408f0a0 tx=0x7f21e4091270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:48.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089700 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f21e408f0a0 tx=0x7f21e4091270).stop 2021-03-09T09:19:48.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:48.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089700 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:48.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:48.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:48.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:48.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e80 con 0x7f21cc005e30 2021-03-09T09:19:48.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f800 con 0x7f21c4005da0 2021-03-09T09:19:48.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:48.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00fe50 con 0x7f21ec0a7f50 2021-03-09T09:19:48.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:48.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:48.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:48.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:48.383 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10535 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:48.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:48.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:48.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:48.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:48.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:19:48.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052540 con 0x7f21ec0a7f50 2021-03-09T09:19:48.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.381+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02eff0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 rx=0x7f21cc030790 tx=0x7f21e40241d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:48.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21ec0a7f50 2021-03-09T09:19:48.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21ec0a7f50 2021-03-09T09:19:48.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:19:48.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.389+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:48.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.389+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:48.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:48.389+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:49.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:19:49.199 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:19:53.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:53.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:53.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02eff0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 rx=0x7f21cc030790 tx=0x7f21e40241d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:53.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02eff0 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 rx=0x7f21cc030790 tx=0x7f21e40241d0).stop 2021-03-09T09:19:53.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02eff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:53.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02eff0 unknown :-1 s=CLOSED pgs=3207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:53.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:53.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:53.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:53.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00fe50 con 0x7f21cc005e30 2021-03-09T09:19:53.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093bb0 con 0x7f21c4005da0 2021-03-09T09:19:53.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009020 con 0x7f21cc083d30 2021-03-09T09:19:53.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:53.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:53.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:53.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:53.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:53.387 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10571 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:53.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:53.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:53.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:53.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:53.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015b70 con 0x7f21cc083d30 2021-03-09T09:19:53.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.381+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:19:53.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092390 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f21cc021070 tx=0x7f21dc003870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:53.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:19:53.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:19:53.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc083d30 2021-03-09T09:19:53.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.389+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:53.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.389+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:53.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:53.389+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:19:58.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc092390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:58.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc092390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:58.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092390 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f21cc021070 tx=0x7f21dc003870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:58.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092390 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f21cc021070 tx=0x7f21dc003870).stop 2021-03-09T09:19:58.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc092390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:58.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092390 unknown :-1 s=CLOSED pgs=3217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:58.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:58.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:58.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:58.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b150 con 0x7f21cc005e30 2021-03-09T09:19:58.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21c4005da0 2021-03-09T09:19:58.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b670 con 0x7f21ec0a7f50 2021-03-09T09:19:58.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:19:58.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:19:58.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:58.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:58.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:58.388 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:19:58.380+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10607 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:19:58.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:58.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:58.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:58.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:58.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:19:58.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:19:58.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02efd0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7f21e0090c90 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:58.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.388+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21ec0a7f50 2021-03-09T09:19:58.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.388+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a3ea0 con 0x7f21ec0a7f50 2021-03-09T09:19:58.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:19:58.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.392+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:19:58.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.392+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:19:58.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:19:58.392+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:03.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:03.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:03.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02efd0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7f21e0090c90 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:03.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02efd0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7f21e0090c90 tx=0x7f21e0003740).stop 2021-03-09T09:20:03.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02efd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:03.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02efd0 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:03.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:03.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:03.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:03.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d040 con 0x7f21cc005e30 2021-03-09T09:20:03.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dba0 con 0x7f21c4005da0 2021-03-09T09:20:03.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21cc083d30 2021-03-09T09:20:03.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:03.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:03.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:03.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:03.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:03.392 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10643 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:03.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b150 con 0x7f21cc083d30 2021-03-09T09:20:03.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:20:03.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.388+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092000 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7f21e00a6270 tx=0x7f21e0039430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:03.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.388+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:20:03.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.388+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc083d30 2021-03-09T09:20:03.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a9000 con 0x7f21cc083d30 2021-03-09T09:20:03.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.392+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:03.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.392+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:03.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:03.392+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:08.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc092000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:08.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc092000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:08.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092000 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7f21e00a6270 tx=0x7f21e0039430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:08.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092000 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7f21e00a6270 tx=0x7f21e0039430).stop 2021-03-09T09:20:08.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc092000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:08.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc092000 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:08.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:08.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:08.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:08.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:08.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010040 con 0x7f21cc005e30 2021-03-09T09:20:08.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:08.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007770 con 0x7f21c4005da0 2021-03-09T09:20:08.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21ec0a7f50 2021-03-09T09:20:08.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:08.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:08.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:08.393 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10684 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:08.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:08.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:08.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:08.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016370 con 0x7f21ec0a7f50 2021-03-09T09:20:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:20:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.388+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089700 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7f21e003cf30 tx=0x7f21e003ce50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.388+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21ec0a7f50 2021-03-09T09:20:08.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.388+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0037b10 con 0x7f21ec0a7f50 2021-03-09T09:20:08.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21ec0a7f50 2021-03-09T09:20:08.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.392+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:08.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.396+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:08.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:08.396+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:13.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:13.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:13.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089700 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7f21e003cf30 tx=0x7f21e003ce50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:13.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089700 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7f21e003cf30 tx=0x7f21e003ce50).stop 2021-03-09T09:20:13.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089700 unknown :-1 s=CLOSED pgs=3245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:13.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e5f0 con 0x7f21cc005e30 2021-03-09T09:20:13.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057fe0 con 0x7f21c4005da0 2021-03-09T09:20:13.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08db10 con 0x7f21cc083d30 2021-03-09T09:20:13.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.384+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:13.395 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10712 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:13.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:13.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:20:13.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016370 con 0x7f21cc083d30 2021-03-09T09:20:13.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.388+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022d20 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f21e0091fe0 tx=0x7f21e0098000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000a1e0 con 0x7f21cc083d30 2021-03-09T09:20:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0037b10 con 0x7f21cc083d30 2021-03-09T09:20:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc083d30 2021-03-09T09:20:13.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.392+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:13.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.396+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:13.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:13.396+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:18.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:18.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:18.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022d20 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f21e0091fe0 tx=0x7f21e0098000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:18.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022d20 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f21e0091fe0 tx=0x7f21e0098000).stop 2021-03-09T09:20:18.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:18.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022d20 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:18.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:18.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:18.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:18.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dd40 con 0x7f21cc005e30 2021-03-09T09:20:18.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21c4005da0 2021-03-09T09:20:18.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0233b0 con 0x7f21ec0a7f50 2021-03-09T09:20:18.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:18.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:18.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:18.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:18.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:18.398 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10752 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:18.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:18.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:18.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:18.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:18.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:20:18.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015b70 con 0x7f21ec0a7f50 2021-03-09T09:20:18.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091f30 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f21dc0530b0 tx=0x7f21dc015a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:18.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010e50 con 0x7f21ec0a7f50 2021-03-09T09:20:18.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f0d0 con 0x7f21ec0a7f50 2021-03-09T09:20:18.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.400+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21ec0a7f50 2021-03-09T09:20:18.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.400+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:18.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.400+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:18.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:18.400+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:19.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:20:19.249 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:20:23.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:23.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:23.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091f30 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f21dc0530b0 tx=0x7f21dc015a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:23.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091f30 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f21dc0530b0 tx=0x7f21dc015a90).stop 2021-03-09T09:20:23.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc091f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:23.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091f30 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:23.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:23.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:23.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:23.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016590 con 0x7f21cc005e30 2021-03-09T09:20:23.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055c60 con 0x7f21c4005da0 2021-03-09T09:20:23.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c990 con 0x7f21cc083d30 2021-03-09T09:20:23.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:23.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:23.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:23.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:23.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:23.401 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:23.392+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10785 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:23.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:23.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:23.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:23.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:23.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057fe0 con 0x7f21cc083d30 2021-03-09T09:20:23.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:20:23.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022990 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f21e0014290 tx=0x7f21e00090c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:23.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:20:23.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002aea0 con 0x7f21cc083d30 2021-03-09T09:20:23.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.404+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc083d30 2021-03-09T09:20:23.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.404+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:23.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.408+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:23.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:23.408+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:28.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:20:28.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:28.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022990 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f21e0014290 tx=0x7f21e00090c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:28.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022990 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f21e0014290 tx=0x7f21e00090c0).stop 2021-03-09T09:20:28.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:28.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022990 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:28.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:28.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:28.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:28.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008090 con 0x7f21cc005e30 2021-03-09T09:20:28.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21c4005da0 2021-03-09T09:20:28.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049110 con 0x7f21ec0a7f50 2021-03-09T09:20:28.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:28.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:28.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:28.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:28.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:28.403 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10823 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:28.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:28.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:28.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:28.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:28.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00fff0 con 0x7f21ec0a7f50 2021-03-09T09:20:28.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.396+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:20:28.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091ce0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:28.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.400+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007da0 con 0x7f21ec0a7f50 2021-03-09T09:20:28.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.404+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:20:28.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.408+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc007da0 con 0x7f21ec0a7f50 2021-03-09T09:20:28.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.408+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:28.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.408+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:28.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:28.408+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:33.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc091ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:33.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc091ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:33.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091ce0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:33.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091ce0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc077de0).stop 2021-03-09T09:20:33.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc091ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:33.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc091ce0 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:33.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:33.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:33.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:33.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc005e30 2021-03-09T09:20:33.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21c4005da0 2021-03-09T09:20:33.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e5f0 con 0x7f21cc083d30 2021-03-09T09:20:33.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:33.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:33.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:33.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:33.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:33.407 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10856 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:33.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:33.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:33.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:33.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:33.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:20:33.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00fff0 con 0x7f21cc083d30 2021-03-09T09:20:33.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.400+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022920 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f21dc0038e0 tx=0x7f21dc01f9d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:33.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.404+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc026420 con 0x7f21cc083d30 2021-03-09T09:20:33.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.408+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc026420 con 0x7f21cc083d30 2021-03-09T09:20:33.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.408+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21cc083d30 2021-03-09T09:20:33.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.408+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:33.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.408+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:33.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:33.408+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:38.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:38.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:38.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022920 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f21dc0038e0 tx=0x7f21dc01f9d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:38.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022920 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f21dc0038e0 tx=0x7f21dc01f9d0).stop 2021-03-09T09:20:38.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:38.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022920 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:38.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:38.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:38.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:38.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc005e30 2021-03-09T09:20:38.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21c4005da0 2021-03-09T09:20:38.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055db0 con 0x7f21ec0a7f50 2021-03-09T09:20:38.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:38.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:38.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:38.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:38.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:38.408 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10900 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:38.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:38.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:38.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:38.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:38.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.399+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:20:38.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.403+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:20:38.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.403+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fb40 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f21e0013460 tx=0x7f21e0098ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:38.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21ec0a7f50 2021-03-09T09:20:38.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21ec0a7f50 2021-03-09T09:20:38.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:20:38.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.411+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:38.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.411+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:38.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:38.411+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:43.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:43.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:43.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fb40 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f21e0013460 tx=0x7f21e0098ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:43.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fb40 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f21e0013460 tx=0x7f21e0098ca0).stop 2021-03-09T09:20:43.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:43.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fb40 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:43.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:43.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:43.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:43.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc005e30 2021-03-09T09:20:43.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004cd0 con 0x7f21c4005da0 2021-03-09T09:20:43.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015f30 con 0x7f21cc083d30 2021-03-09T09:20:43.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:43.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:43.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:43.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:43.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:43.411 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10939 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:43.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:43.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:43.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:43.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:43.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057fe0 con 0x7f21cc083d30 2021-03-09T09:20:43.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:20:43.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022720 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc01fc80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:43.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005da0 con 0x7f21cc083d30 2021-03-09T09:20:43.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062dd0 con 0x7f21cc083d30 2021-03-09T09:20:43.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055d80 con 0x7f21cc083d30 2021-03-09T09:20:43.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.415+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:43.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.415+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:43.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:43.415+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:48.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:48.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:48.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022720 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc01fc80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:48.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022720 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc01fc80).stop 2021-03-09T09:20:48.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:48.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022720 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:48.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:48.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:48.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:48.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e70 con 0x7f21cc005e30 2021-03-09T09:20:48.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0235d0 con 0x7f21c4005da0 2021-03-09T09:20:48.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:48.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0528f0 con 0x7f21ec0a7f50 2021-03-09T09:20:48.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:48.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:48.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:48.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:48.413 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:48.407+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10975 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:48.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:48.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:48.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:48.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:48.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:20:48.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:20:48.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04faa0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f21e007d5d0 tx=0x7f21e00835d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:48.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21ec0a7f50 2021-03-09T09:20:48.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21ec0a7f50 2021-03-09T09:20:48.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0098d10 con 0x7f21ec0a7f50 2021-03-09T09:20:48.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.419+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:48.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.419+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:48.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:48.419+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:49.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:20:49.295 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:20:53.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.407+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:53.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.407+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:53.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.407+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04faa0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f21e007d5d0 tx=0x7f21e00835d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:53.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.407+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04faa0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f21e007d5d0 tx=0x7f21e00835d0).stop 2021-03-09T09:20:53.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.407+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04faa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:53.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04faa0 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:53.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:53.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:53.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:53.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0235d0 con 0x7f21cc005e30 2021-03-09T09:20:53.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02bc30 con 0x7f21c4005da0 2021-03-09T09:20:53.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008090 con 0x7f21cc083d30 2021-03-09T09:20:53.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:53.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:53.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:53.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:53.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:53.416 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11009 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:53.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:53.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:53.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:53.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:53.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015f30 con 0x7f21cc083d30 2021-03-09T09:20:53.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:20:53.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00ca90 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7f21e0087d80 tx=0x7f21e002afc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:53.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc083d30 2021-03-09T09:20:53.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc083d30 2021-03-09T09:20:53.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc083d30 2021-03-09T09:20:53.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.419+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:53.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.419+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:53.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:53.419+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:20:58.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:58.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:58.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00ca90 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7f21e0087d80 tx=0x7f21e002afc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:58.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00ca90 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7f21e0087d80 tx=0x7f21e002afc0).stop 2021-03-09T09:20:58.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00ca90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:58.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00ca90 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:58.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:58.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:58.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:58.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004910 con 0x7f21cc005e30 2021-03-09T09:20:58.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0235d0 con 0x7f21c4005da0 2021-03-09T09:20:58.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e7a0 con 0x7f21ec0a7f50 2021-03-09T09:20:58.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:20:58.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:58.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:58.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:20:58.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.411+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:58.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:58.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:58.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:58.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:58.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f4d0 con 0x7f21ec0a7f50 2021-03-09T09:20:58.419 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11041 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:20:58.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:20:58.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa30 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e407f230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:58.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f3a0 con 0x7f21ec0a7f50 2021-03-09T09:20:58.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21ec0a7f50 2021-03-09T09:20:58.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e403e080 con 0x7f21ec0a7f50 2021-03-09T09:20:58.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.419+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:20:58.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.419+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:20:58.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:20:58.419+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:03.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:03.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:03.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa30 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e407f230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:03.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa30 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e407f230).stop 2021-03-09T09:21:03.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:03.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa30 unknown :-1 s=CLOSED pgs=3352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:03.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:03.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:03.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:03.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21cc005e30 2021-03-09T09:21:03.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0235d0 con 0x7f21c4005da0 2021-03-09T09:21:03.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011e50 con 0x7f21cc083d30 2021-03-09T09:21:03.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:03.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:03.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:03.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:03.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:03.421 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11073 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:03.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:03.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:03.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:03.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:03.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024140 con 0x7f21cc083d30 2021-03-09T09:21:03.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057fe0 con 0x7f21cc083d30 2021-03-09T09:21:03.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c9e0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7f21cc00c960 tx=0x7f21dc069120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:03.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc083d30 2021-03-09T09:21:03.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010ec0 con 0x7f21cc083d30 2021-03-09T09:21:03.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.427+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21cc083d30 2021-03-09T09:21:03.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.427+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:03.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.427+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:03.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:03.427+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:08.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:21:08.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:08.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c9e0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7f21cc00c960 tx=0x7f21dc069120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:08.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c9e0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7f21cc00c960 tx=0x7f21dc069120).stop 2021-03-09T09:21:08.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00c9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:08.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c9e0 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:08.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:08.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:08.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:08.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21cc005e30 2021-03-09T09:21:08.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0578e0 con 0x7f21c4005da0 2021-03-09T09:21:08.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:21:08.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:08.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:08.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:08.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.415+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:08.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:08.423 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11121 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:08.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:08.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:08.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:08.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:08.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:21:08.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024140 con 0x7f21ec0a7f50 2021-03-09T09:21:08.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f950 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7f21cc02eb70 tx=0x7f21dc02e610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:08.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005190 con 0x7f21ec0a7f50 2021-03-09T09:21:08.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005190 con 0x7f21ec0a7f50 2021-03-09T09:21:08.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.427+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06d000 con 0x7f21ec0a7f50 2021-03-09T09:21:08.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.427+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:08.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.427+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:08.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:08.427+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:13.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:21:13.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:13.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f950 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7f21cc02eb70 tx=0x7f21dc02e610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:13.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f950 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7f21cc02eb70 tx=0x7f21dc02e610).stop 2021-03-09T09:21:13.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:13.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f950 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:13.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:13.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:13.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:13.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21cc005e30 2021-03-09T09:21:13.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d9c0 con 0x7f21c4005da0 2021-03-09T09:21:13.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:21:13.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:13.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:13.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:13.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:13.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.419+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:13.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:13.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:13.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:13.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:13.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0133e0 con 0x7f21cc083d30 2021-03-09T09:21:13.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015f30 con 0x7f21cc083d30 2021-03-09T09:21:13.429 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11149 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:13.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c990 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 rx=0x7f21e00a6270 tx=0x7f21e002a170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:13.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc083d30 2021-03-09T09:21:13.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002ada0 con 0x7f21cc083d30 2021-03-09T09:21:13.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc083d30 2021-03-09T09:21:13.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.431+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:13.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.431+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:13.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:13.431+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:18.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:18.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:18.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c990 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 rx=0x7f21e00a6270 tx=0x7f21e002a170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:18.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c990 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 rx=0x7f21e00a6270 tx=0x7f21e002a170).stop 2021-03-09T09:21:18.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc00c990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:18.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc00c990 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:18.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:18.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:18.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:18.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012220 con 0x7f21cc005e30 2021-03-09T09:21:18.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023770 con 0x7f21c4005da0 2021-03-09T09:21:18.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:21:18.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:18.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:18.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:18.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:18.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:18.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:18.430 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11189 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:18.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:18.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:18.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:18.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f310 con 0x7f21ec0a7f50 2021-03-09T09:21:18.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f4d0 con 0x7f21ec0a7f50 2021-03-09T09:21:18.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.423+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050800 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e4053f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:18.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.427+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040090 con 0x7f21ec0a7f50 2021-03-09T09:21:18.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.427+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403e070 con 0x7f21ec0a7f50 2021-03-09T09:21:18.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21ec0a7f50 2021-03-09T09:21:18.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.431+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:18.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.435+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:18.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:18.435+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:19.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:21:19.364 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:21:23.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:21:23.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:23.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050800 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e4053f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:23.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050800 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e4053f10).stop 2021-03-09T09:21:23.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:23.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050800 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:23.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:23.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:23.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0220a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:23.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21cc005e30 2021-03-09T09:21:23.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21c4005da0 2021-03-09T09:21:23.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012220 con 0x7f21cc083d30 2021-03-09T09:21:23.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:23.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:23.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:23.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.422+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:23.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:23.431 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11216 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:23.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:23.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:23.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:23.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:23.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057fe0 con 0x7f21cc083d30 2021-03-09T09:21:23.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:21:23.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0220a0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 rx=0x7f21dc06d500 tx=0x7f21dc069120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:23.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:21:23.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21cc083d30 2021-03-09T09:21:23.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc083d30 2021-03-09T09:21:23.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.434+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:23.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.434+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:23.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:23.434+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:28.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:21:28.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:28.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0220a0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 rx=0x7f21dc06d500 tx=0x7f21dc069120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:28.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0220a0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 rx=0x7f21dc06d500 tx=0x7f21dc069120).stop 2021-03-09T09:21:28.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0220a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:28.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0220a0 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:28.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:28.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:28.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:28.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21cc005e30 2021-03-09T09:21:28.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21c4005da0 2021-03-09T09:21:28.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7c0 con 0x7f21ec0a7f50 2021-03-09T09:21:28.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:28.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:28.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:28.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:28.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.426+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:28.434 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11254 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:28.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:28.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:28.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:28.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:28.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:21:28.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0133e0 con 0x7f21ec0a7f50 2021-03-09T09:21:28.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f230 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 rx=0x7f21e00a97d0 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:28.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21ec0a7f50 2021-03-09T09:21:28.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a95d0 con 0x7f21ec0a7f50 2021-03-09T09:21:28.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21ec0a7f50 2021-03-09T09:21:28.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.434+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:28.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.438+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:28.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:28.438+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:33.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:33.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f230 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 rx=0x7f21e00a97d0 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f230 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 rx=0x7f21e00a97d0 tx=0x7f21e000b3e0).stop 2021-03-09T09:21:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f230 unknown :-1 s=CLOSED pgs=3416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:33.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:33.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc021eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:33.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:33.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21cc005e30 2021-03-09T09:21:33.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0892a0 con 0x7f21c4005da0 2021-03-09T09:21:33.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0513d0 con 0x7f21cc083d30 2021-03-09T09:21:33.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:33.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:33.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:33.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:33.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc021eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:33.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:33.438 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11301 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:33.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:33.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:33.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:33.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c510 con 0x7f21cc083d30 2021-03-09T09:21:33.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.430+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f310 con 0x7f21cc083d30 2021-03-09T09:21:33.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc021eb0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 rx=0x7f21cc050750 tx=0x7f21e40b1120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:33.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4015a30 con 0x7f21cc083d30 2021-03-09T09:21:33.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1350 con 0x7f21cc083d30 2021-03-09T09:21:33.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e403e070 con 0x7f21cc083d30 2021-03-09T09:21:33.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.438+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:33.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.442+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:33.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:33.442+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:38.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc021eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:21:38.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc021eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:38.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc021eb0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 rx=0x7f21cc050750 tx=0x7f21e40b1120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:38.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc021eb0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 rx=0x7f21cc050750 tx=0x7f21e40b1120).stop 2021-03-09T09:21:38.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc021eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:38.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc021eb0 unknown :-1 s=CLOSED pgs=3426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:38.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:38.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:38.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21cc005e30 2021-03-09T09:21:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024430 con 0x7f21c4005da0 2021-03-09T09:21:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7c0 con 0x7f21ec0a7f50 2021-03-09T09:21:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:38.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:38.439 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11335 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:38.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:38.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024140 con 0x7f21ec0a7f50 2021-03-09T09:21:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c510 con 0x7f21ec0a7f50 2021-03-09T09:21:38.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f090 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4036d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:38.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:21:38.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403d810 con 0x7f21ec0a7f50 2021-03-09T09:21:38.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007160 con 0x7f21ec0a7f50 2021-03-09T09:21:38.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.442+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:38.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.442+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:38.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:38.442+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:43.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:43.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:43.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f090 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4036d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:43.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f090 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4036d30).stop 2021-03-09T09:21:43.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:43.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f090 unknown :-1 s=CLOSED pgs=3437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:43.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:43.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:43.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc044fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:43.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21cc005e30 2021-03-09T09:21:43.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d300 con 0x7f21c4005da0 2021-03-09T09:21:43.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7c0 con 0x7f21cc083d30 2021-03-09T09:21:43.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:43.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:43.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc044fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:43.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:43.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:43.442 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:43.434+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11369 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:43.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:43.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:43.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:43.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:43.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:21:43.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057fe0 con 0x7f21cc083d30 2021-03-09T09:21:43.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc044fd0 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 rx=0x7f21dc06fa40 tx=0x7f21dc06f6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:43.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc083d30 2021-03-09T09:21:43.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21cc083d30 2021-03-09T09:21:43.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062c40 con 0x7f21cc083d30 2021-03-09T09:21:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.442+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.442+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:43.442+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:48.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc044fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:21:48.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc044fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:48.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc044fd0 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 rx=0x7f21dc06fa40 tx=0x7f21dc06f6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:48.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc044fd0 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 rx=0x7f21dc06fa40 tx=0x7f21dc06f6c0).stop 2021-03-09T09:21:48.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc044fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:48.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc044fd0 unknown :-1 s=CLOSED pgs=3448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:48.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:48.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:48.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:48.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21cc005e30 2021-03-09T09:21:48.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e970 con 0x7f21c4005da0 2021-03-09T09:21:48.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f790 con 0x7f21ec0a7f50 2021-03-09T09:21:48.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:48.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:48.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:48.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:48.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:48.444 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11405 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:48.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:48.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:48.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:48.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:48.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:21:48.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.438+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024140 con 0x7f21ec0a7f50 2021-03-09T09:21:48.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.442+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021fb0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 rx=0x7f21e4013000 tx=0x7f21e4028190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:48.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21ec0a7f50 2021-03-09T09:21:48.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024410 con 0x7f21ec0a7f50 2021-03-09T09:21:48.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21ec0a7f50 2021-03-09T09:21:48.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.446+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:48.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.446+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:48.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:48.446+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:49.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:21:49.416 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:21:53.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:53.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:53.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021fb0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 rx=0x7f21e4013000 tx=0x7f21e4028190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:53.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021fb0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 rx=0x7f21e4013000 tx=0x7f21e4028190).stop 2021-03-09T09:21:53.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:53.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021fb0 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:53.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:53.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:53.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:53.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21cc005e30 2021-03-09T09:21:53.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016890 con 0x7f21c4005da0 2021-03-09T09:21:53.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051390 con 0x7f21cc083d30 2021-03-09T09:21:53.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:53.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:53.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:53.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:53.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:53.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:53.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:53.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:53.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:53.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d4a0 con 0x7f21cc083d30 2021-03-09T09:21:53.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:21:53.454 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11443 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:53.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 rx=0x7f21cc015e90 tx=0x7f21dc020560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:53.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:21:53.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc04d890 con 0x7f21cc083d30 2021-03-09T09:21:53.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc083d30 2021-03-09T09:21:53.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.450+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:53.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.450+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:53.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:53.450+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:21:58.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:21:58.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:58.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 rx=0x7f21cc015e90 tx=0x7f21dc020560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:58.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 rx=0x7f21cc015e90 tx=0x7f21dc020560).stop 2021-03-09T09:21:58.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:58.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=3469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:58.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:58.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:58.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:58.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf30 con 0x7f21cc005e30 2021-03-09T09:21:58.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016890 con 0x7f21c4005da0 2021-03-09T09:21:58.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7c0 con 0x7f21ec0a7f50 2021-03-09T09:21:58.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:21:58.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:58.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:21:58.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:58.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:58.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:58.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:58.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:58.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:58.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057fe0 con 0x7f21ec0a7f50 2021-03-09T09:21:58.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:21:58.451 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:21:58.442+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11471 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:21:58.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.446+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f510 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e000b4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:58.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21ec0a7f50 2021-03-09T09:21:58.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:21:58.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:21:58.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.450+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:21:58.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.450+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:21:58.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:21:58.450+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:03.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:03.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:03.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f510 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e000b4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:03.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f510 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e000b4d0).stop 2021-03-09T09:22:03.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:03.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f510 unknown :-1 s=CLOSED pgs=3480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:03.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:03.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:03.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:03.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051390 con 0x7f21cc005e30 2021-03-09T09:22:03.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc090dc0 con 0x7f21c4005da0 2021-03-09T09:22:03.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016890 con 0x7f21cc083d30 2021-03-09T09:22:03.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:03.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:03.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:03.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:03.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:03.452 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11504 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:03.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:03.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:03.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:03.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:03.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0133e0 con 0x7f21cc083d30 2021-03-09T09:22:03.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:22:03.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e40ac3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:03.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.454+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040080 con 0x7f21cc083d30 2021-03-09T09:22:03.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.454+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40ac0d0 con 0x7f21cc083d30 2021-03-09T09:22:03.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.458+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21cc083d30 2021-03-09T09:22:03.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.458+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:03.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.458+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:03.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:03.458+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:08.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:08.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:08.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e40ac3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:08.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e40ac3c0).stop 2021-03-09T09:22:08.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:08.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:08.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:08.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0222b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:08.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:08.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052780 con 0x7f21cc005e30 2021-03-09T09:22:08.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016890 con 0x7f21c4005da0 2021-03-09T09:22:08.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051390 con 0x7f21ec0a7f50 2021-03-09T09:22:08.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:08.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:08.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:08.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:08.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:08.454 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11551 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:08.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:08.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:08.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:08.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:08.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c510 con 0x7f21ec0a7f50 2021-03-09T09:22:08.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0133e0 con 0x7f21ec0a7f50 2021-03-09T09:22:08.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0222b0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 rx=0x7f21e4027f00 tx=0x7f21e4017fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:08.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21ec0a7f50 2021-03-09T09:22:08.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a41a0 con 0x7f21ec0a7f50 2021-03-09T09:22:08.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21ec0a7f50 2021-03-09T09:22:08.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.457+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:08.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.457+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:08.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:08.457+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:13.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0222b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:13.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0222b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:13.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0222b0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 rx=0x7f21e4027f00 tx=0x7f21e4017fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:13.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0222b0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 rx=0x7f21e4027f00 tx=0x7f21e4017fa0).stop 2021-03-09T09:22:13.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0222b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:13.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0222b0 unknown :-1 s=CLOSED pgs=3503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:13.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:13.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:13.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:13.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc090f60 con 0x7f21cc005e30 2021-03-09T09:22:13.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf10 con 0x7f21c4005da0 2021-03-09T09:22:13.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0451b0 con 0x7f21cc083d30 2021-03-09T09:22:13.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:13.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:13.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:13.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:13.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:13.457 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11576 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:13.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:13.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:13.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:13.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:13.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:22:13.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.449+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c510 con 0x7f21cc083d30 2021-03-09T09:22:13.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e40b44d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:13.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21cc083d30 2021-03-09T09:22:13.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024bc0 con 0x7f21cc083d30 2021-03-09T09:22:13.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21cc083d30 2021-03-09T09:22:13.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.457+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:13.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.461+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:13.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:13.461+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:18.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:18.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:18.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e40b44d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:18.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 rx=0x7f21e40279f0 tx=0x7f21e40b44d0).stop 2021-03-09T09:22:18.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:18.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=3512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:18.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:18.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:18.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:18.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e230 con 0x7f21cc005e30 2021-03-09T09:22:18.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf10 con 0x7f21c4005da0 2021-03-09T09:22:18.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f790 con 0x7f21ec0a7f50 2021-03-09T09:22:18.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:18.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:18.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:18.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:18.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:18.459 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11614 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:18.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:18.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:18.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:18.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:18.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:22:18.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:22:18.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013050 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:18.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0039500 con 0x7f21ec0a7f50 2021-03-09T09:22:18.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0039500 con 0x7f21ec0a7f50 2021-03-09T09:22:18.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21ec0a7f50 2021-03-09T09:22:18.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.465+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:18.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:18.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:18.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:19.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:22:19.459 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:22:23.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:23.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:23.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013050 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:23.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013050 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e007c5a0).stop 2021-03-09T09:22:23.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc013050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:23.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc013050 unknown :-1 s=CLOSED pgs=3527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:23.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:23.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:23.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:23.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cf10 con 0x7f21cc005e30 2021-03-09T09:22:23.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0451b0 con 0x7f21c4005da0 2021-03-09T09:22:23.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f790 con 0x7f21cc083d30 2021-03-09T09:22:23.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:23.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:23.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:23.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:23.462 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:23.453+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11658 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:23.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:23.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:23.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:23.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:23.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:23.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:22:23.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:22:23.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0217c0 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 rx=0x7f21e00a3580 tx=0x7f21e00a3660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:23.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:22:23.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0081eb0 con 0x7f21cc083d30 2021-03-09T09:22:23.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00263c0 con 0x7f21cc083d30 2021-03-09T09:22:23.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.465+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:23.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:23.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:23.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:28.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0217c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:28.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0217c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:28.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0217c0 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 rx=0x7f21e00a3580 tx=0x7f21e00a3660).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:28.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0217c0 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 rx=0x7f21e00a3580 tx=0x7f21e00a3660).stop 2021-03-09T09:22:28.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0217c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:28.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0217c0 unknown :-1 s=CLOSED pgs=3537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:28.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:28.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:28.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:28.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f790 con 0x7f21cc005e30 2021-03-09T09:22:28.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0451b0 con 0x7f21c4005da0 2021-03-09T09:22:28.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c530 con 0x7f21ec0a7f50 2021-03-09T09:22:28.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:28.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:28.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:28.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:28.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:28.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:28.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:28.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:28.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:28.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052730 con 0x7f21ec0a7f50 2021-03-09T09:22:28.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21ec0a7f50 2021-03-09T09:22:28.467 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11687 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:28.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089540 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0081f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:28.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a3000 con 0x7f21ec0a7f50 2021-03-09T09:22:28.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0007e40 con 0x7f21ec0a7f50 2021-03-09T09:22:28.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006f90 con 0x7f21ec0a7f50 2021-03-09T09:22:28.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.465+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:28.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:28.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:28.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:33.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:33.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:33.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089540 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0081f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:33.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089540 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0081f20).stop 2021-03-09T09:22:33.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:33.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089540 unknown :-1 s=CLOSED pgs=3544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:33.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:33.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:33.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:33.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092230 con 0x7f21cc005e30 2021-03-09T09:22:33.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0451b0 con 0x7f21c4005da0 2021-03-09T09:22:33.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f790 con 0x7f21cc083d30 2021-03-09T09:22:33.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:33.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:33.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:33.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:33.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:33.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:33.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:33.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:33.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:33.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc083d30 2021-03-09T09:22:33.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:22:33.468 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11720 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:33.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012f70 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4007540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:33.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21cc083d30 2021-03-09T09:22:33.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027c20 con 0x7f21cc083d30 2021-03-09T09:22:33.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21cc083d30 2021-03-09T09:22:33.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.465+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:33.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:33.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:33.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:38.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc012f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:22:38.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc012f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:38.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012f70 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4007540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:38.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012f70 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4007540).stop 2021-03-09T09:22:38.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc012f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:38.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012f70 unknown :-1 s=CLOSED pgs=3554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:38.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:38.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:38.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:38.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f7b0 con 0x7f21cc005e30 2021-03-09T09:22:38.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092230 con 0x7f21c4005da0 2021-03-09T09:22:38.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011df0 con 0x7f21ec0a7f50 2021-03-09T09:22:38.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:38.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:38.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:38.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:38.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:38.470 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11764 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:38.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:38.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:38.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:38.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:38.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c510 con 0x7f21ec0a7f50 2021-03-09T09:22:38.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.461+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21ec0a7f50 2021-03-09T09:22:38.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894f0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 rx=0x7f21e40ac250 tx=0x7f21e40053d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:38.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040090 con 0x7f21ec0a7f50 2021-03-09T09:22:38.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:22:38.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:22:38.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.469+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:38.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:38.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:38.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:43.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:43.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:43.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894f0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 rx=0x7f21e40ac250 tx=0x7f21e40053d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:43.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894f0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 rx=0x7f21e40ac250 tx=0x7f21e40053d0).stop 2021-03-09T09:22:43.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:43.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894f0 unknown :-1 s=CLOSED pgs=3565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:43.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:43.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:43.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:43.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d290 con 0x7f21cc005e30 2021-03-09T09:22:43.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0451b0 con 0x7f21c4005da0 2021-03-09T09:22:43.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011df0 con 0x7f21cc083d30 2021-03-09T09:22:43.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:43.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:43.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:43.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:43.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:43.473 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11802 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:43.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:43.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:43.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:43.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:43.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:22:43.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c510 con 0x7f21cc083d30 2021-03-09T09:22:43.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.469+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012ed0 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 rx=0x7f21e4027aa0 tx=0x7f21e4027b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:43.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040040 con 0x7f21cc083d30 2021-03-09T09:22:43.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1ec0 con 0x7f21cc083d30 2021-03-09T09:22:43.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc083d30 2021-03-09T09:22:43.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.477+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:43.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:43.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:43.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:48.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc012ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:22:48.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc012ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:48.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012ed0 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 rx=0x7f21e4027aa0 tx=0x7f21e4027b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:48.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012ed0 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 rx=0x7f21e4027aa0 tx=0x7f21e4027b90).stop 2021-03-09T09:22:48.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc012ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:48.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc012ed0 unknown :-1 s=CLOSED pgs=3577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:48.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:48.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:48.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:48.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c7b0 con 0x7f21cc005e30 2021-03-09T09:22:48.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08f790 con 0x7f21c4005da0 2021-03-09T09:22:48.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0451b0 con 0x7f21ec0a7f50 2021-03-09T09:22:48.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:48.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:48.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:48.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:48.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:48.475 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11831 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:48.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:48.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:48.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:48.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:48.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:22:48.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052730 con 0x7f21ec0a7f50 2021-03-09T09:22:48.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0039430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:48.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21ec0a7f50 2021-03-09T09:22:48.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21ec0a7f50 2021-03-09T09:22:48.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21ec0a7f50 2021-03-09T09:22:48.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.477+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:48.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:48.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:48.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:49.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:22:49.497 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:22:53.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:53.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:53.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0039430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:53.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0039430).stop 2021-03-09T09:22:53.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:53.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 unknown :-1 s=CLOSED pgs=3586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:53.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:53.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c7b0 con 0x7f21cc005e30 2021-03-09T09:22:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097650 con 0x7f21c4005da0 2021-03-09T09:22:53.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f10 con 0x7f21cc083d30 2021-03-09T09:22:53.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:53.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:53.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:53.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:53.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:53.478 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11863 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:53.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:53.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:53.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086120 con 0x7f21cc083d30 2021-03-09T09:22:53.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:22:53.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0523a0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 rx=0x7f21e4042060 tx=0x7f21e40b00b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:53.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:22:53.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1ec0 con 0x7f21cc083d30 2021-03-09T09:22:53.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402b640 con 0x7f21cc083d30 2021-03-09T09:22:53.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.476+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:53.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:53.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:53.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:22:58.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:58.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:58.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0523a0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 rx=0x7f21e4042060 tx=0x7f21e40b00b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:58.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0523a0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 rx=0x7f21e4042060 tx=0x7f21e40b00b0).stop 2021-03-09T09:22:58.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0523a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:58.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0523a0 unknown :-1 s=CLOSED pgs=3597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:58.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:58.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:58.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0091f0 con 0x7f21cc005e30 2021-03-09T09:22:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097690 con 0x7f21c4005da0 2021-03-09T09:22:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0912a0 con 0x7f21ec0a7f50 2021-03-09T09:22:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:22:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:22:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:58.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:58.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:58.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:58.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:58.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c510 con 0x7f21ec0a7f50 2021-03-09T09:22:58.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21ec0a7f50 2021-03-09T09:22:58.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 rx=0x7f21e003c510 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:58.482 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11913 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:22:58.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21ec0a7f50 2021-03-09T09:22:58.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21ec0a7f50 2021-03-09T09:22:58.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21ec0a7f50 2021-03-09T09:22:58.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.480+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:22:58.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:22:58.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:22:58.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:03.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:03.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:03.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 rx=0x7f21e003c510 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:03.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 rx=0x7f21e003c510 tx=0x7f21e0003820).stop 2021-03-09T09:23:03.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0894a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:03.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0894a0 unknown :-1 s=CLOSED pgs=3608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043c90 con 0x7f21cc005e30 2021-03-09T09:23:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d290 con 0x7f21c4005da0 2021-03-09T09:23:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0526a0 con 0x7f21cc083d30 2021-03-09T09:23:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:03.483 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11946 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:03.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:03.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:03.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091ec0 con 0x7f21cc083d30 2021-03-09T09:23:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c510 con 0x7f21cc083d30 2021-03-09T09:23:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 rx=0x7f21e003c510 tx=0x7f21e003d980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:03.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:23:03.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0014ec0 con 0x7f21cc083d30 2021-03-09T09:23:03.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc083d30 2021-03-09T09:23:03.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.484+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:03.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.484+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:03.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:03.484+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:08.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:08.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:08.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 rx=0x7f21e003c510 tx=0x7f21e003d980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:08.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 rx=0x7f21e003c510 tx=0x7f21e003d980).stop 2021-03-09T09:23:08.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:08.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:08.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:08.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:08.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:08.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc005e30 2021-03-09T09:23:08.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21c4005da0 2021-03-09T09:23:08.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033ac0 con 0x7f21ec0a7f50 2021-03-09T09:23:08.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:08.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:08.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:08.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:08.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:08.486 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11981 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:08.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:08.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:08.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:08.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:08.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc033c30 con 0x7f21ec0a7f50 2021-03-09T09:23:08.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d4a0 con 0x7f21ec0a7f50 2021-03-09T09:23:08.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0526a0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7f21dc01d040 tx=0x7f21dc069120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:08.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02e430 con 0x7f21ec0a7f50 2021-03-09T09:23:08.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:23:08.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f080 con 0x7f21ec0a7f50 2021-03-09T09:23:08.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.488+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:08.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:08.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:08.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:13.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:23:13.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:13.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0526a0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7f21dc01d040 tx=0x7f21dc069120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:13.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0526a0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7f21dc01d040 tx=0x7f21dc069120).stop 2021-03-09T09:23:13.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0526a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:13.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0526a0 unknown :-1 s=CLOSED pgs=3636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:13.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:13.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:13.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:13.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041530 con 0x7f21cc005e30 2021-03-09T09:23:13.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21c4005da0 2021-03-09T09:23:13.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21cc083d30 2021-03-09T09:23:13.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:13.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:13.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:13.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:13.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:13.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:13.488 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12008 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:13.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:13.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:13.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:13.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012d10 con 0x7f21cc083d30 2021-03-09T09:23:13.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc083d30 2021-03-09T09:23:13.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 rx=0x7f21e003d820 tx=0x7f21e004dec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:13.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:23:13.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004a40 con 0x7f21cc083d30 2021-03-09T09:23:13.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc083d30 2021-03-09T09:23:13.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.488+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:13.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:13.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:13.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:18.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:18.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:18.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 rx=0x7f21e003d820 tx=0x7f21e004dec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:18.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 rx=0x7f21e003d820 tx=0x7f21e004dec0).stop 2021-03-09T09:23:18.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:18.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:18.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:18.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:18.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:18.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc005e30 2021-03-09T09:23:18.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21c4005da0 2021-03-09T09:23:18.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023390 con 0x7f21ec0a7f50 2021-03-09T09:23:18.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:18.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:18.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:18.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:18.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.480+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:18.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:18.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:18.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:18.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:18.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023500 con 0x7f21ec0a7f50 2021-03-09T09:23:18.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:23:18.492 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12051 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:18.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089480 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 rx=0x7f21e405bfd0 tx=0x7f21e40a63e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:18.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091b20 con 0x7f21ec0a7f50 2021-03-09T09:23:18.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403e120 con 0x7f21ec0a7f50 2021-03-09T09:23:18.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027c20 con 0x7f21ec0a7f50 2021-03-09T09:23:18.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.492+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:18.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:18.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:18.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:19.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:23:19.541 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:23:23.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:23.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:23.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089480 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 rx=0x7f21e405bfd0 tx=0x7f21e40a63e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:23.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089480 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 rx=0x7f21e405bfd0 tx=0x7f21e40a63e0).stop 2021-03-09T09:23:23.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:23.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089480 unknown :-1 s=CLOSED pgs=3650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:23.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:23.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:23.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:23.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093ff0 con 0x7f21cc005e30 2021-03-09T09:23:23.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007840 con 0x7f21c4005da0 2021-03-09T09:23:23.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:23:23.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:23.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:23.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:23.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:23.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:23.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:23.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:23.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:23.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:23.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:23:23.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012d10 con 0x7f21cc083d30 2021-03-09T09:23:23.495 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12090 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:23.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e004dd00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:23.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc083d30 2021-03-09T09:23:23.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc083d30 2021-03-09T09:23:23.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009dd80 con 0x7f21cc083d30 2021-03-09T09:23:23.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.492+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:23.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.496+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:23.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:23.496+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:28.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:28.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:28.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e004dd00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:28.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e004dd00).stop 2021-03-09T09:23:28.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:28.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:28.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:28.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:28.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:28.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:23:28.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21c4005da0 2021-03-09T09:23:28.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089480 con 0x7f21ec0a7f50 2021-03-09T09:23:28.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:28.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:28.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:28.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:28.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:28.497 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12123 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:28.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:28.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:28.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:28.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:28.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c510 con 0x7f21ec0a7f50 2021-03-09T09:23:28.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc033c30 con 0x7f21ec0a7f50 2021-03-09T09:23:28.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012b70 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 rx=0x7f21cc03db80 tx=0x7f21dc006c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:28.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010970 con 0x7f21ec0a7f50 2021-03-09T09:23:28.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010970 con 0x7f21ec0a7f50 2021-03-09T09:23:28.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21ec0a7f50 2021-03-09T09:23:28.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.500+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:28.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:28.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:28.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:33.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:33.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:33.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012b70 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 rx=0x7f21cc03db80 tx=0x7f21dc006c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:33.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012b70 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 rx=0x7f21cc03db80 tx=0x7f21dc006c50).stop 2021-03-09T09:23:33.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc012b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:33.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc012b70 unknown :-1 s=CLOSED pgs=3676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:33.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:33.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:33.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:33.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d000 con 0x7f21cc005e30 2021-03-09T09:23:33.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089480 con 0x7f21c4005da0 2021-03-09T09:23:33.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092350 con 0x7f21cc083d30 2021-03-09T09:23:33.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:33.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:33.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:33.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:33.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:33.499 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12158 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:33.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:33.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:33.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:33.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:33.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:23:33.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023500 con 0x7f21cc083d30 2021-03-09T09:23:33.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4017fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:33.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4040070 con 0x7f21cc083d30 2021-03-09T09:23:33.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091a50 con 0x7f21cc083d30 2021-03-09T09:23:33.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091a50 con 0x7f21cc083d30 2021-03-09T09:23:33.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.500+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:33.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:33.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:33.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:38.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:23:38.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:38.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4017fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:38.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4017fc0).stop 2021-03-09T09:23:38.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:38.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:38.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:38.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:38.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:38.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d000 con 0x7f21cc005e30 2021-03-09T09:23:38.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:38.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092350 con 0x7f21c4005da0 2021-03-09T09:23:38.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ea0 con 0x7f21ec0a7f50 2021-03-09T09:23:38.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:38.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:38.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:38.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:38.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:38.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:38.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:38.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:38.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc033b10 con 0x7f21ec0a7f50 2021-03-09T09:23:38.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c510 con 0x7f21ec0a7f50 2021-03-09T09:23:38.504 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12201 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:38.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0129b0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 rx=0x7f21dc009380 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:38.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc060140 con 0x7f21ec0a7f50 2021-03-09T09:23:38.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05cdf0 con 0x7f21ec0a7f50 2021-03-09T09:23:38.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc006ff0 con 0x7f21ec0a7f50 2021-03-09T09:23:38.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.504+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:38.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.504+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:38.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:38.504+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:43.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0129b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:23:43.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0129b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:43.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0129b0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 rx=0x7f21dc009380 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:43.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0129b0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 rx=0x7f21dc009380 tx=0x7f21dc009290).stop 2021-03-09T09:23:43.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0129b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:43.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0129b0 unknown :-1 s=CLOSED pgs=3693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:43.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:43.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:43.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:43.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021e80 con 0x7f21cc005e30 2021-03-09T09:23:43.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d000 con 0x7f21c4005da0 2021-03-09T09:23:43.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ea0 con 0x7f21cc083d30 2021-03-09T09:23:43.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:43.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:43.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:43.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:43.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:43.509 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12237 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:43.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:43.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:43.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:43.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:43.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0526a0 con 0x7f21cc083d30 2021-03-09T09:23:43.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc033b10 con 0x7f21cc083d30 2021-03-09T09:23:43.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 rx=0x7f21cc0912a0 tx=0x7f21dc068d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:43.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068e30 con 0x7f21cc083d30 2021-03-09T09:23:43.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc060140 con 0x7f21cc083d30 2021-03-09T09:23:43.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069c20 con 0x7f21cc083d30 2021-03-09T09:23:43.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.503+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:43.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.503+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:43.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:43.503+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:48.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:23:48.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:48.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 rx=0x7f21cc0912a0 tx=0x7f21dc068d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:48.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 rx=0x7f21cc0912a0 tx=0x7f21dc068d30).stop 2021-03-09T09:23:48.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:48.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:48.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:48.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:48.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:48.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc033cb0 con 0x7f21cc005e30 2021-03-09T09:23:48.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0578d0 con 0x7f21c4005da0 2021-03-09T09:23:48.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012950 con 0x7f21ec0a7f50 2021-03-09T09:23:48.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:48.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:48.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:48.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.499+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:48.507 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12276 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:48.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:48.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:48.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:48.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:48.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:48.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012a90 con 0x7f21ec0a7f50 2021-03-09T09:23:48.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0526a0 con 0x7f21ec0a7f50 2021-03-09T09:23:48.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0573e0 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 rx=0x7f21cc004f80 tx=0x7f21dc091d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:48.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068e30 con 0x7f21ec0a7f50 2021-03-09T09:23:48.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc068e30 con 0x7f21ec0a7f50 2021-03-09T09:23:48.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062000 con 0x7f21ec0a7f50 2021-03-09T09:23:48.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.511+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:48.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.515+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:48.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:48.515+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:49.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:23:49.610 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:23:53.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:53.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:53.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0573e0 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 rx=0x7f21cc004f80 tx=0x7f21dc091d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:53.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0573e0 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 rx=0x7f21cc004f80 tx=0x7f21dc091d50).stop 2021-03-09T09:23:53.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0573e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:53.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0573e0 unknown :-1 s=CLOSED pgs=3717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:53.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:53.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:53.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:53.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050630 con 0x7f21cc005e30 2021-03-09T09:23:53.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0578d0 con 0x7f21c4005da0 2021-03-09T09:23:53.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0526a0 con 0x7f21cc083d30 2021-03-09T09:23:53.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:53.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:53.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:53.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:53.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:53.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:53.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:53.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:53.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:53.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0527e0 con 0x7f21cc083d30 2021-03-09T09:23:53.511 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:53.503+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:53.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:23:53.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 rx=0x7f21e00375f0 tx=0x7f21e004dc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:53.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0090e90 con 0x7f21cc083d30 2021-03-09T09:23:53.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0090e90 con 0x7f21cc083d30 2021-03-09T09:23:53.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009dd80 con 0x7f21cc083d30 2021-03-09T09:23:53.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.515+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:53.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.515+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:53.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:53.515+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:23:58.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:58.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:58.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 rx=0x7f21e00375f0 tx=0x7f21e004dc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:58.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 rx=0x7f21e00375f0 tx=0x7f21e004dc90).stop 2021-03-09T09:23:58.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:58.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:58.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:58.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:58.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:58.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0526a0 con 0x7f21cc005e30 2021-03-09T09:23:58.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050630 con 0x7f21c4005da0 2021-03-09T09:23:58.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089e20 con 0x7f21ec0a7f50 2021-03-09T09:23:58.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:58.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:23:58.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:58.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:23:58.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:58.512 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12344 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:23:58.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:58.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:58.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:58.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:58.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:23:58.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0527e0 con 0x7f21ec0a7f50 2021-03-09T09:23:58.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057370 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e001f150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:58.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21ec0a7f50 2021-03-09T09:23:58.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21ec0a7f50 2021-03-09T09:23:58.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0004070 con 0x7f21ec0a7f50 2021-03-09T09:23:58.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.519+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:23:58.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:23:58.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:23:58.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:03.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:03.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:03.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057370 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e001f150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:03.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057370 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e001f150).stop 2021-03-09T09:24:03.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:03.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057370 unknown :-1 s=CLOSED pgs=3736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:03.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:03.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:03.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:03.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0304c0 con 0x7f21cc005e30 2021-03-09T09:24:03.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050b70 con 0x7f21c4005da0 2021-03-09T09:24:03.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0526a0 con 0x7f21cc083d30 2021-03-09T09:24:03.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:03.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:03.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:03.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:03.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.507+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:03.515 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12374 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:03.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:03.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:03.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:03.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:03.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0527e0 con 0x7f21cc083d30 2021-03-09T09:24:03.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:24:03.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.511+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 rx=0x7f21e001fa60 tx=0x7f21e001f730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:03.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc083d30 2021-03-09T09:24:03.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc083d30 2021-03-09T09:24:03.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0004070 con 0x7f21cc083d30 2021-03-09T09:24:03.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.519+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:03.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:03.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:03.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:08.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:08.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:08.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 rx=0x7f21e001fa60 tx=0x7f21e001f730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:08.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 rx=0x7f21e001fa60 tx=0x7f21e001f730).stop 2021-03-09T09:24:08.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:08.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:08.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:08.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:08.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:08.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc005e30 2021-03-09T09:24:08.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:08.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045360 con 0x7f21c4005da0 2021-03-09T09:24:08.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:08.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0572d0 con 0x7f21ec0a7f50 2021-03-09T09:24:08.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:08.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:08.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:08.527 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:08.511+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12403 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:08.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:08.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:08.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:08.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:08.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057440 con 0x7f21ec0a7f50 2021-03-09T09:24:08.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012a90 con 0x7f21ec0a7f50 2021-03-09T09:24:08.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ce00 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 rx=0x7f21dc00af50 tx=0x7f21dc07cc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:08.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc091be0 con 0x7f21ec0a7f50 2021-03-09T09:24:08.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:24:08.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05cb00 con 0x7f21ec0a7f50 2021-03-09T09:24:08.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.523+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:08.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.523+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:08.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:08.523+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:13.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:13.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:13.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ce00 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 rx=0x7f21dc00af50 tx=0x7f21dc07cc20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:13.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ce00 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 rx=0x7f21dc00af50 tx=0x7f21dc07cc20).stop 2021-03-09T09:24:13.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ce00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:13.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ce00 unknown :-1 s=CLOSED pgs=3757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:13.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:13.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:13.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:13.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc005e30 2021-03-09T09:24:13.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057280 con 0x7f21c4005da0 2021-03-09T09:24:13.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0526a0 con 0x7f21cc083d30 2021-03-09T09:24:13.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:13.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:13.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:13.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:13.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:13.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:13.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:13.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:13.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:13.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0527e0 con 0x7f21cc083d30 2021-03-09T09:24:13.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:24:13.523 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12440 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:13.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 rx=0x7f21e4020060 tx=0x7f21e4007240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:13.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003ad0 con 0x7f21cc083d30 2021-03-09T09:24:13.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403e120 con 0x7f21cc083d30 2021-03-09T09:24:13.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40919f0 con 0x7f21cc083d30 2021-03-09T09:24:13.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.519+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:13.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.523+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:13.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:13.523+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:18.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:24:18.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:18.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 rx=0x7f21e4020060 tx=0x7f21e4007240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:18.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 rx=0x7f21e4020060 tx=0x7f21e4007240).stop 2021-03-09T09:24:18.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0521b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:18.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0521b0 unknown :-1 s=CLOSED pgs=3768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:18.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:18.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:18.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:18.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:18.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0526a0 con 0x7f21cc005e30 2021-03-09T09:24:18.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:18.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086500 con 0x7f21c4005da0 2021-03-09T09:24:18.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:24:18.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:18.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:18.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:18.524 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12488 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:18.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:18.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:18.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:18.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:18.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:24:18.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057440 con 0x7f21ec0a7f50 2021-03-09T09:24:18.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ca70 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 rx=0x7f21dc0627e0 tx=0x7f21dc062280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:18.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062620 con 0x7f21ec0a7f50 2021-03-09T09:24:18.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062620 con 0x7f21ec0a7f50 2021-03-09T09:24:18.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc091dc0 con 0x7f21ec0a7f50 2021-03-09T09:24:18.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.531+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:18.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.531+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:18.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:18.531+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:19.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:24:19.648 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:24:23.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:23.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:23.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ca70 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 rx=0x7f21dc0627e0 tx=0x7f21dc062280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:23.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ca70 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 rx=0x7f21dc0627e0 tx=0x7f21dc062280).stop 2021-03-09T09:24:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ca70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ca70 unknown :-1 s=CLOSED pgs=3778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:23.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:23.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022a00 con 0x7f21cc005e30 2021-03-09T09:24:23.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4005da0 2021-03-09T09:24:23.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b150 con 0x7f21cc083d30 2021-03-09T09:24:23.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:23.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:23.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:23.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:23.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:23.526 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12522 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:23.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:23.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:23.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:23.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:23.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0521b0 con 0x7f21cc083d30 2021-03-09T09:24:23.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:24:23.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057280 secure :-1 s=READY pgs=3789 cs=0 l=1 rev1=1 rx=0x7f21cc02f370 tx=0x7f21dc026af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:23.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00af80 con 0x7f21cc083d30 2021-03-09T09:24:23.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00af80 con 0x7f21cc083d30 2021-03-09T09:24:23.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21cc083d30 2021-03-09T09:24:23.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.530+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:23.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.530+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:23.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:23.530+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:28.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:28.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:28.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057280 secure :-1 s=READY pgs=3789 cs=0 l=1 rev1=1 rx=0x7f21cc02f370 tx=0x7f21dc026af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:28.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057280 secure :-1 s=READY pgs=3789 cs=0 l=1 rev1=1 rx=0x7f21cc02f370 tx=0x7f21dc026af0).stop 2021-03-09T09:24:28.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057280 unknown :-1 s=CLOSED pgs=3789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013270 con 0x7f21cc005e30 2021-03-09T09:24:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052680 con 0x7f21c4005da0 2021-03-09T09:24:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022a00 con 0x7f21ec0a7f50 2021-03-09T09:24:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:28.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:28.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:28.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:28.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:28.529 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12560 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:28.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:28.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21ec0a7f50 2021-03-09T09:24:28.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0521b0 con 0x7f21ec0a7f50 2021-03-09T09:24:28.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c9d0 secure :-1 s=READY pgs=3801 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc026df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:28.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21ec0a7f50 2021-03-09T09:24:28.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc076e40 con 0x7f21ec0a7f50 2021-03-09T09:24:28.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21ec0a7f50 2021-03-09T09:24:28.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.530+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:28.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.534+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:28.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:28.534+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:33.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:33.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:33.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c9d0 secure :-1 s=READY pgs=3801 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc026df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:33.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c9d0 secure :-1 s=READY pgs=3801 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc026df0).stop 2021-03-09T09:24:33.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:33.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c9d0 unknown :-1 s=CLOSED pgs=3801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:33.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:33.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:33.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:33.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:33.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:33.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:33.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:33.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:33.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013270 con 0x7f21cc005e30 2021-03-09T09:24:33.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0233c0 con 0x7f21c4005da0 2021-03-09T09:24:33.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ccc0 con 0x7f21cc083d30 2021-03-09T09:24:33.532 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12597 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:33.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:33.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:33.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:33.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:33.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:24:33.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:24:33.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 rx=0x7f21cc02cfc0 tx=0x7f21dc062120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:33.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc083d30 2021-03-09T09:24:33.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06f650 con 0x7f21cc083d30 2021-03-09T09:24:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003960 con 0x7f21cc083d30 2021-03-09T09:24:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.530+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.534+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:33.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:33.534+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:38.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:38.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:38.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 rx=0x7f21cc02cfc0 tx=0x7f21dc062120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:38.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 rx=0x7f21cc02cfc0 tx=0x7f21dc062120).stop 2021-03-09T09:24:38.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:38.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=CLOSED pgs=3810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:38.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:38.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:38.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:38.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:38.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:24:38.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:38.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cc70 con 0x7f21c4005da0 2021-03-09T09:24:38.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057e20 con 0x7f21ec0a7f50 2021-03-09T09:24:38.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:38.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:38.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:38.535 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12632 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:38.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:38.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:38.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:38.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:38.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057f60 con 0x7f21ec0a7f50 2021-03-09T09:24:38.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:24:38.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c920 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0b5980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:38.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21ec0a7f50 2021-03-09T09:24:38.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:24:38.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc026b60 con 0x7f21ec0a7f50 2021-03-09T09:24:38.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.538+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:38.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.538+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:38.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:38.538+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:43.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:43.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:43.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c920 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0b5980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:43.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c920 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0b5980).stop 2021-03-09T09:24:43.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:43.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c920 unknown :-1 s=CLOSED pgs=3821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:43.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:43.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:43.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:43.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dbd0 con 0x7f21cc005e30 2021-03-09T09:24:43.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011ec0 con 0x7f21c4005da0 2021-03-09T09:24:43.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc083d30 2021-03-09T09:24:43.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:43.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:43.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:43.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:43.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:43.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:43.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:43.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:43.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:43.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc083d30 2021-03-09T09:24:43.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057f60 con 0x7f21cc083d30 2021-03-09T09:24:43.539 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:43.530+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12670 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:43.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 rx=0x7f21dc085d90 tx=0x7f21dc06f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:43.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:24:43.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21cc083d30 2021-03-09T09:24:43.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21cc083d30 2021-03-09T09:24:43.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.538+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:43.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.542+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:43.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:43.542+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:48.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:48.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:48.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 rx=0x7f21dc085d90 tx=0x7f21dc06f700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:48.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 rx=0x7f21dc085d90 tx=0x7f21dc06f700).stop 2021-03-09T09:24:48.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:48.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=CLOSED pgs=3832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:48.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:48.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:48.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:48.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057f60 con 0x7f21cc005e30 2021-03-09T09:24:48.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:48.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:48.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0914c0 con 0x7f21c4005da0 2021-03-09T09:24:48.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091600 con 0x7f21ec0a7f50 2021-03-09T09:24:48.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:48.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:48.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:48.541 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12701 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:48.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:48.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:48.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:48.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc0317a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:48.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cd80 con 0x7f21ec0a7f50 2021-03-09T09:24:48.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:24:48.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 rx=0x7f21dc00b110 tx=0x7f21dc018f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:48.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00a780 con 0x7f21ec0a7f50 2021-03-09T09:24:48.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00a5a0 con 0x7f21ec0a7f50 2021-03-09T09:24:48.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21ec0a7f50 2021-03-09T09:24:48.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.542+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:48.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.542+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:48.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:48.542+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:49.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:24:49.796 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:24:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 msgr2=0x7f21cc03ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:24:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 msgr2=0x7f21cc03ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 0x7f21cc03ef70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc006d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:51.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 0x7f21cc03ef70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc006d70).stop 2021-03-09T09:24:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:24:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21c40033c0 2021-03-09T09:24:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T09:24:51.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c40033c0 0x7f21cc03ef70 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:51.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:51.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:51.982+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e001f730).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:53.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:53.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:53.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 rx=0x7f21dc00b110 tx=0x7f21dc018f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:53.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 rx=0x7f21dc00b110 tx=0x7f21dc018f40).stop 2021-03-09T09:24:53.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:53.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:53.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:53.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:53.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:53.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048f70 con 0x7f21cc005e30 2021-03-09T09:24:53.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094090 con 0x7f21c4005da0 2021-03-09T09:24:53.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:53.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21c40033c0 2021-03-09T09:24:53.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:53.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:53.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:53.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:53.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:53.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:53.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:53.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:53.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0575c0 con 0x7f21c40033c0 2021-03-09T09:24:53.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cd80 con 0x7f21c40033c0 2021-03-09T09:24:53.546 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12739 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:53.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0317a0 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 rx=0x7f21dc017de0 tx=0x7f21dc003870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:53.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21c40033c0 2021-03-09T09:24:53.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc097e20 con 0x7f21c40033c0 2021-03-09T09:24:53.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05f070 con 0x7f21c40033c0 2021-03-09T09:24:53.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.546+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:53.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.546+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:53.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:53.546+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:24:58.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:58.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:58.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0317a0 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 rx=0x7f21dc017de0 tx=0x7f21dc003870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:58.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0317a0 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 rx=0x7f21dc017de0 tx=0x7f21dc003870).stop 2021-03-09T09:24:58.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0317a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:58.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0317a0 unknown :-1 s=CLOSED pgs=3853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:58.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:58.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:58.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:58.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048f70 con 0x7f21cc005e30 2021-03-09T09:24:58.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0130f0 con 0x7f21c4005da0 2021-03-09T09:24:58.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:24:58.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f170 con 0x7f21ec0a7f50 2021-03-09T09:24:58.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:58.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:24:58.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:58.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.538+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:58.547 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12776 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:24:58.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:58.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:58.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:58.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:58.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:24:58.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0527e0 con 0x7f21ec0a7f50 2021-03-09T09:24:58.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.542+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 rx=0x7f21cc056930 tx=0x7f21e4003a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:58.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091bd0 con 0x7f21ec0a7f50 2021-03-09T09:24:58.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e10 con 0x7f21ec0a7f50 2021-03-09T09:24:58.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091460 con 0x7f21ec0a7f50 2021-03-09T09:24:58.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.550+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:24:58.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.550+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:24:58.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:24:58.550+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:03.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:03.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:03.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 rx=0x7f21cc056930 tx=0x7f21e4003a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:03.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 rx=0x7f21cc056930 tx=0x7f21e4003a10).stop 2021-03-09T09:25:03.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:03.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:03.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:03.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:03.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:03.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048f70 con 0x7f21cc005e30 2021-03-09T09:25:03.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031440 con 0x7f21c4005da0 2021-03-09T09:25:03.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031580 con 0x7f21c40033c0 2021-03-09T09:25:03.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:03.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:03.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:03.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:03.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:03.553 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:03.541+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12811 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:03.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:03.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:03.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:03.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:03.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0316f0 con 0x7f21c40033c0 2021-03-09T09:25:03.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21c40033c0 2021-03-09T09:25:03.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 rx=0x7f21cc03d2c0 tx=0x7f21e40a5ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:03.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21c40033c0 2021-03-09T09:25:03.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027c00 con 0x7f21c40033c0 2021-03-09T09:25:03.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21c40033c0 2021-03-09T09:25:03.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.553+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:03.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.553+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:03.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:03.553+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:08.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:08.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:08.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 rx=0x7f21cc03d2c0 tx=0x7f21e40a5ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:08.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 rx=0x7f21cc03d2c0 tx=0x7f21e40a5ec0).stop 2021-03-09T09:25:08.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:08.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=3875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:08.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:08.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:08.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:08.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e7c0 con 0x7f21cc005e30 2021-03-09T09:25:08.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048f70 con 0x7f21c4005da0 2021-03-09T09:25:08.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:25:08.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:08.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:08.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:08.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:08.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:08.553 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12848 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:08.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:08.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:08.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:08.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:08.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031880 con 0x7f21ec0a7f50 2021-03-09T09:25:08.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0316f0 con 0x7f21ec0a7f50 2021-03-09T09:25:08.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.549+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3885 cs=0 l=1 rev1=1 rx=0x7f21e4055ef0 tx=0x7f21e4055b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:08.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:25:08.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017280 con 0x7f21ec0a7f50 2021-03-09T09:25:08.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:25:08.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.553+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:08.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.553+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:08.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:08.553+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:13.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:13.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:13.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3885 cs=0 l=1 rev1=1 rx=0x7f21e4055ef0 tx=0x7f21e4055b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:13.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3885 cs=0 l=1 rev1=1 rx=0x7f21e4055ef0 tx=0x7f21e4055b70).stop 2021-03-09T09:25:13.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:13.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:13.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:13.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:13.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:13.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094af0 con 0x7f21cc005e30 2021-03-09T09:25:13.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21c4005da0 2021-03-09T09:25:13.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024f30 con 0x7f21c40033c0 2021-03-09T09:25:13.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:13.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:13.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:13.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.545+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:13.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:13.556 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12885 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:13.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:13.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:13.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:13.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:13.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025070 con 0x7f21c40033c0 2021-03-09T09:25:13.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0575c0 con 0x7f21c40033c0 2021-03-09T09:25:13.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 rx=0x7f21cc059180 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:13.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08aec0 con 0x7f21c40033c0 2021-03-09T09:25:13.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc097e20 con 0x7f21c40033c0 2021-03-09T09:25:13.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc006b90 con 0x7f21c40033c0 2021-03-09T09:25:13.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.557+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:13.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.557+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:13.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:13.557+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:18.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:18.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:18.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 rx=0x7f21cc059180 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:18.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 rx=0x7f21cc059180 tx=0x7f21dc02dfc0).stop 2021-03-09T09:25:18.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:18.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=3896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:18.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:18.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:18.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:18.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024f30 con 0x7f21cc005e30 2021-03-09T09:25:18.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21c4005da0 2021-03-09T09:25:18.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030690 con 0x7f21ec0a7f50 2021-03-09T09:25:18.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:18.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:18.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:18.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:18.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:18.559 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:18.549+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12920 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:18.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:18.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:18.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:18.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:18.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0307d0 con 0x7f21ec0a7f50 2021-03-09T09:25:18.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025070 con 0x7f21ec0a7f50 2021-03-09T09:25:18.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:18.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21ec0a7f50 2021-03-09T09:25:18.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06b3c0 con 0x7f21ec0a7f50 2021-03-09T09:25:18.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:25:18.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.565+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:18.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.565+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:18.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:18.565+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:19.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:25:19.859 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:25:23.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:23.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:23.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:23.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc07ff90).stop 2021-03-09T09:25:23.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:23.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:23.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:23.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:23.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:23.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e7d0 con 0x7f21cc005e30 2021-03-09T09:25:23.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024f30 con 0x7f21c4005da0 2021-03-09T09:25:23.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025070 con 0x7f21c40033c0 2021-03-09T09:25:23.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:23.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:23.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:23.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:23.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:23.561 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12958 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:23.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:23.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:23.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:23.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:23.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0251e0 con 0x7f21c40033c0 2021-03-09T09:25:23.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031880 con 0x7f21c40033c0 2021-03-09T09:25:23.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.553+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 rx=0x7f21e4029040 tx=0x7f21e40414a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:23.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21c40033c0 2021-03-09T09:25:23.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4028120 con 0x7f21c40033c0 2021-03-09T09:25:23.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055ca0 con 0x7f21c40033c0 2021-03-09T09:25:23.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.561+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:23.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.565+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:23.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:23.565+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:28.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:28.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:28.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 rx=0x7f21e4029040 tx=0x7f21e40414a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:28.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 rx=0x7f21e4029040 tx=0x7f21e40414a0).stop 2021-03-09T09:25:28.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:28.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=3918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:28.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:28.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:28.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:28.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0316c0 con 0x7f21cc005e30 2021-03-09T09:25:28.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031800 con 0x7f21c4005da0 2021-03-09T09:25:28.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031940 con 0x7f21ec0a7f50 2021-03-09T09:25:28.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:28.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:28.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:28.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:28.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:28.564 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12979 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:28.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:28.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:28.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:28.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:28.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031ab0 con 0x7f21ec0a7f50 2021-03-09T09:25:28.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0307d0 con 0x7f21ec0a7f50 2021-03-09T09:25:28.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 rx=0x7f21cc03dcc0 tx=0x7f21dc06b430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:28.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06b000 con 0x7f21ec0a7f50 2021-03-09T09:25:28.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01f250 con 0x7f21ec0a7f50 2021-03-09T09:25:28.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc007a20 con 0x7f21ec0a7f50 2021-03-09T09:25:28.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.565+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:28.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.569+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:28.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:28.569+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:33.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:33.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:33.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 rx=0x7f21cc03dcc0 tx=0x7f21dc06b430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:33.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 rx=0x7f21cc03dcc0 tx=0x7f21dc06b430).stop 2021-03-09T09:25:33.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:33.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:33.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:33.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:33.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:33.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051290 con 0x7f21cc005e30 2021-03-09T09:25:33.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0316c0 con 0x7f21c4005da0 2021-03-09T09:25:33.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031800 con 0x7f21c40033c0 2021-03-09T09:25:33.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:33.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:33.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:33.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:33.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.557+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:33.567 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13027 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:33.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:33.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:33.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:33.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:33.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:25:33.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0251e0 con 0x7f21c40033c0 2021-03-09T09:25:33.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 rx=0x7f21cc082f10 tx=0x7f21e40039a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:33.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40092d0 con 0x7f21c40033c0 2021-03-09T09:25:33.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091e30 con 0x7f21c40033c0 2021-03-09T09:25:33.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40092d0 con 0x7f21c40033c0 2021-03-09T09:25:33.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.569+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:33.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.569+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:33.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:33.569+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:38.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:38.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:38.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 rx=0x7f21cc082f10 tx=0x7f21e40039a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:38.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 rx=0x7f21cc082f10 tx=0x7f21e40039a0).stop 2021-03-09T09:25:38.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:38.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=3941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:38.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:38.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:38.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:38.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031940 con 0x7f21cc005e30 2021-03-09T09:25:38.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d8a0 con 0x7f21c4005da0 2021-03-09T09:25:38.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e230 con 0x7f21ec0a7f50 2021-03-09T09:25:38.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:38.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:38.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:38.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:38.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:38.570 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13061 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:38.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:38.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:38.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:38.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:38.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04e370 con 0x7f21ec0a7f50 2021-03-09T09:25:38.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031ab0 con 0x7f21ec0a7f50 2021-03-09T09:25:38.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 rx=0x7f21dc068de0 tx=0x7f21dc06b3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:38.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06def0 con 0x7f21ec0a7f50 2021-03-09T09:25:38.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:25:38.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc007a20 con 0x7f21ec0a7f50 2021-03-09T09:25:38.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.569+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:38.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:38.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:38.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:43.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:43.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:43.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 rx=0x7f21dc068de0 tx=0x7f21dc06b3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:43.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 rx=0x7f21dc068de0 tx=0x7f21dc06b3c0).stop 2021-03-09T09:25:43.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:43.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:43.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:43.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:43.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:43.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dcc0 con 0x7f21cc005e30 2021-03-09T09:25:43.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08de00 con 0x7f21c4005da0 2021-03-09T09:25:43.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08df40 con 0x7f21c40033c0 2021-03-09T09:25:43.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:43.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:43.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:43.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:43.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:43.572 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:43.565+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13099 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:43.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:43.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:43.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:43.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:43.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031940 con 0x7f21c40033c0 2021-03-09T09:25:43.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04e370 con 0x7f21c40033c0 2021-03-09T09:25:43.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 rx=0x7f21dc005c60 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:43.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:25:43.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017160 con 0x7f21c40033c0 2021-03-09T09:25:43.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc01f760 con 0x7f21c40033c0 2021-03-09T09:25:43.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.573+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:43.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.577+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:43.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:43.577+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:48.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:48.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:48.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 rx=0x7f21dc005c60 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:48.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 rx=0x7f21dc005c60 tx=0x7f21dc07ff90).stop 2021-03-09T09:25:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=3963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:48.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:48.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e280 con 0x7f21cc005e30 2021-03-09T09:25:48.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e3c0 con 0x7f21c4005da0 2021-03-09T09:25:48.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e500 con 0x7f21ec0a7f50 2021-03-09T09:25:48.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:48.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:48.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:48.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:48.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.564+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:48.575 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13131 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:48.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:48.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:48.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:48.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:48.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08db20 con 0x7f21ec0a7f50 2021-03-09T09:25:48.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031940 con 0x7f21ec0a7f50 2021-03-09T09:25:48.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc08af00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:48.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21ec0a7f50 2021-03-09T09:25:48.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc004110 con 0x7f21ec0a7f50 2021-03-09T09:25:48.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:25:48.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.580+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:48.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:48.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:48.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:49.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:25:49.918 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:25:53.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:53.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:53.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc08af00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:53.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc08af00).stop 2021-03-09T09:25:53.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:53.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:53.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:53.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:53.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:53.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031ae0 con 0x7f21cc005e30 2021-03-09T09:25:53.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e280 con 0x7f21c4005da0 2021-03-09T09:25:53.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e3c0 con 0x7f21c40033c0 2021-03-09T09:25:53.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:53.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:53.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:53.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:53.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:53.577 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:53.568+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13166 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:53.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:53.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:53.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:53.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:53.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0215f0 con 0x7f21c40033c0 2021-03-09T09:25:53.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:25:53.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.572+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 rx=0x7f21cc016ff0 tx=0x7f21e40093b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:53.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091c40 con 0x7f21c40033c0 2021-03-09T09:25:53.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21c40033c0 2021-03-09T09:25:53.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4091460 con 0x7f21c40033c0 2021-03-09T09:25:53.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.580+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:53.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:53.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:53.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:25:58.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:58.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:58.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 rx=0x7f21cc016ff0 tx=0x7f21e40093b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:58.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 rx=0x7f21cc016ff0 tx=0x7f21e40093b0).stop 2021-03-09T09:25:58.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:58.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=3983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:58.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03da70 con 0x7f21cc005e30 2021-03-09T09:25:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T09:25:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cdb0 con 0x7f21ec0a7f50 2021-03-09T09:25:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:25:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:25:58.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:58.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:58.581 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13205 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:25:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04e500 con 0x7f21ec0a7f50 2021-03-09T09:25:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08db20 con 0x7f21ec0a7f50 2021-03-09T09:25:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.572+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 rx=0x7f21cc04cf80 tx=0x7f21dc009340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0705b0 con 0x7f21ec0a7f50 2021-03-09T09:25:58.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06dc30 con 0x7f21ec0a7f50 2021-03-09T09:25:58.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06dc30 con 0x7f21ec0a7f50 2021-03-09T09:25:58.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.580+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:25:58.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:25:58.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:25:58.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:03.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:03.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:03.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 rx=0x7f21cc04cf80 tx=0x7f21dc009340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:03.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 rx=0x7f21cc04cf80 tx=0x7f21dc009340).stop 2021-03-09T09:26:03.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:03.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01c890 unknown :-1 s=CLOSED pgs=3992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:03.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:03.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:03.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:03.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03da70 con 0x7f21cc005e30 2021-03-09T09:26:03.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T09:26:03.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cdb0 con 0x7f21c40033c0 2021-03-09T09:26:03.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:03.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:03.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:03.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:03.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:03.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:03.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:03.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:03.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:03.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04e500 con 0x7f21c40033c0 2021-03-09T09:26:03.584 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13243 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:03.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08db20 con 0x7f21c40033c0 2021-03-09T09:26:03.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.576+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 rx=0x7f21cc013190 tx=0x7f21dc01ffb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:03.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004850 con 0x7f21c40033c0 2021-03-09T09:26:03.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0049b0 con 0x7f21c40033c0 2021-03-09T09:26:03.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004850 con 0x7f21c40033c0 2021-03-09T09:26:03.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.584+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:03.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.584+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:03.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:03.584+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:08.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:08.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:08.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 rx=0x7f21cc013190 tx=0x7f21dc01ffb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:08.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 rx=0x7f21cc013190 tx=0x7f21dc01ffb0).stop 2021-03-09T09:26:08.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:08.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=4003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:08.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:08.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:08.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:08.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023150 con 0x7f21cc005e30 2021-03-09T09:26:08.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03da70 con 0x7f21c4005da0 2021-03-09T09:26:08.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056660 con 0x7f21ec0a7f50 2021-03-09T09:26:08.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:08.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:08.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:08.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:08.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:08.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:08.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:08.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:08.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:08.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:26:08.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08db20 con 0x7f21ec0a7f50 2021-03-09T09:26:08.587 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13277 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:08.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ede0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc091070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:08.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21ec0a7f50 2021-03-09T09:26:08.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085c40 con 0x7f21ec0a7f50 2021-03-09T09:26:08.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06d000 con 0x7f21ec0a7f50 2021-03-09T09:26:08.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.588+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:08.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.588+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:08.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:08.588+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:13.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:13.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:13.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ede0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc091070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:13.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ede0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc091070).stop 2021-03-09T09:26:13.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ede0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:13.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ede0 unknown :-1 s=CLOSED pgs=4014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:13.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:13.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:13.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:13.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b1a0 con 0x7f21cc005e30 2021-03-09T09:26:13.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089ec0 con 0x7f21c4005da0 2021-03-09T09:26:13.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023150 con 0x7f21c40033c0 2021-03-09T09:26:13.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:13.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:13.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:13.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:13.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:13.729 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13314 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00cdb0 con 0x7f21c40033c0 2021-03-09T09:26:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:26:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 rx=0x7f21dc076a80 tx=0x7f21dc050770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b50d0 con 0x7f21c40033c0 2021-03-09T09:26:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:26:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b50d0 con 0x7f21c40033c0 2021-03-09T09:26:13.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.588+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:13.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.588+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:13.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:13.588+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:18.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:18.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:18.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 rx=0x7f21dc076a80 tx=0x7f21dc050770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:18.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 rx=0x7f21dc076a80 tx=0x7f21dc050770).stop 2021-03-09T09:26:18.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:18.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01cd80 unknown :-1 s=CLOSED pgs=4024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:18.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:18.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:18.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:18.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054030 con 0x7f21cc005e30 2021-03-09T09:26:18.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc010370 con 0x7f21c4005da0 2021-03-09T09:26:18.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:26:18.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:18.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:18.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:18.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:18.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:18.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:18.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:18.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:18.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:18.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08dcc0 con 0x7f21ec0a7f50 2021-03-09T09:26:18.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0215f0 con 0x7f21ec0a7f50 2021-03-09T09:26:18.594 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13349 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:18.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.584+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021780 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 rx=0x7f21e4003a50 tx=0x7f21e4091240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:18.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40286a0 con 0x7f21ec0a7f50 2021-03-09T09:26:18.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407dec0 con 0x7f21ec0a7f50 2021-03-09T09:26:18.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.592+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:26:18.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.592+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:18.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.592+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:18.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:18.592+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:19.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:26:19.971 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:26:23.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:23.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:23.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021780 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 rx=0x7f21e4003a50 tx=0x7f21e4091240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:23.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021780 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 rx=0x7f21e4003a50 tx=0x7f21e4091240).stop 2021-03-09T09:26:23.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:23.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021780 unknown :-1 s=CLOSED pgs=4034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:23.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:23.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:23.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:23.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021e10 con 0x7f21cc005e30 2021-03-09T09:26:23.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021600 con 0x7f21c4005da0 2021-03-09T09:26:23.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054030 con 0x7f21c40033c0 2021-03-09T09:26:23.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:23.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:23.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:23.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:23.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:23.599 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13387 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:23.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:23.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:23.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:23.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:23.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b1a0 con 0x7f21c40033c0 2021-03-09T09:26:23.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00cdb0 con 0x7f21c40033c0 2021-03-09T09:26:23.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093960 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc009300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:23.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c40033c0 2021-03-09T09:26:23.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc096ce0 con 0x7f21c40033c0 2021-03-09T09:26:23.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.596+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:26:23.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.596+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:23.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.596+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:23.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:23.596+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:28.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:28.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:28.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093960 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc009300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:28.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093960 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc009300).stop 2021-03-09T09:26:28.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:28.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093960 unknown :-1 s=CLOSED pgs=4046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:28.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:28.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:28.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:28.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054030 con 0x7f21cc005e30 2021-03-09T09:26:28.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ef90 con 0x7f21c4005da0 2021-03-09T09:26:28.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f0d0 con 0x7f21ec0a7f50 2021-03-09T09:26:28.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:28.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:28.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:28.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:28.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:28.599 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13421 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:28.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:28.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:28.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:28.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:28.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:26:28.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b1a0 con 0x7f21ec0a7f50 2021-03-09T09:26:28.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 rx=0x7f21cc01d2c0 tx=0x7f21dc08ce60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:28.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069680 con 0x7f21ec0a7f50 2021-03-09T09:26:28.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21ec0a7f50 2021-03-09T09:26:28.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b58c0 con 0x7f21ec0a7f50 2021-03-09T09:26:28.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.595+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:28.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:28.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:28.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:33.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:33.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:33.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 rx=0x7f21cc01d2c0 tx=0x7f21dc08ce60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:33.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 rx=0x7f21cc01d2c0 tx=0x7f21dc08ce60).stop 2021-03-09T09:26:33.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:33.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:33.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:33.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:33.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:33.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021fc0 con 0x7f21cc005e30 2021-03-09T09:26:33.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0153c0 con 0x7f21c4005da0 2021-03-09T09:26:33.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053e90 con 0x7f21c40033c0 2021-03-09T09:26:33.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:33.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:33.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:33.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:33.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:33.601 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13461 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:33.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:33.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:33.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:33.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:33.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053fd0 con 0x7f21c40033c0 2021-03-09T09:26:33.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:26:33.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc0abfe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:33.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc097470 con 0x7f21c40033c0 2021-03-09T09:26:33.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08ca70 con 0x7f21c40033c0 2021-03-09T09:26:33.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc08ca70 con 0x7f21c40033c0 2021-03-09T09:26:33.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.603+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:33.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.603+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:33.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:33.603+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:38.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:38.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:38.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc0abfe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:38.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc0abfe0).stop 2021-03-09T09:26:38.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:38.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=CLOSED pgs=4067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:38.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:38.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:38.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:38.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054160 con 0x7f21cc005e30 2021-03-09T09:26:38.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f90 con 0x7f21c4005da0 2021-03-09T09:26:38.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ef90 con 0x7f21ec0a7f50 2021-03-09T09:26:38.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:38.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:38.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:38.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:38.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:38.604 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13489 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:38.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:38.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:38.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:38.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:38.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:26:38.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053fd0 con 0x7f21ec0a7f50 2021-03-09T09:26:38.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:38.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc097470 con 0x7f21ec0a7f50 2021-03-09T09:26:38.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc097470 con 0x7f21ec0a7f50 2021-03-09T09:26:38.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b58c0 con 0x7f21ec0a7f50 2021-03-09T09:26:38.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.607+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:38.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:38.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:38.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:43.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:26:43.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:43.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:43.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc02dfc0).stop 2021-03-09T09:26:43.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:43.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:43.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:43.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:43.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:43.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:26:43.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b6a0 con 0x7f21c4005da0 2021-03-09T09:26:43.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0153c0 con 0x7f21c40033c0 2021-03-09T09:26:43.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:43.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:43.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:43.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:43.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:43.610 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13533 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:43.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:43.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:43.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:43.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:43.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:26:43.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086120 con 0x7f21c40033c0 2021-03-09T09:26:43.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0977b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:43.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21c40033c0 2021-03-09T09:26:43.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007eb0 con 0x7f21c40033c0 2021-03-09T09:26:43.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc097d80 con 0x7f21c40033c0 2021-03-09T09:26:43.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.611+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:43.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:43.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:43.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:48.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:48.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:48.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0977b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:48.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0977b0).stop 2021-03-09T09:26:48.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:48.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=CLOSED pgs=4090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:48.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:48.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:48.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:48.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0305f0 con 0x7f21cc005e30 2021-03-09T09:26:48.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21c4005da0 2021-03-09T09:26:48.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e8d0 con 0x7f21ec0a7f50 2021-03-09T09:26:48.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:48.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:48.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:48.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:48.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:48.609 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13567 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:48.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:48.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:48.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:48.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:48.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ef90 con 0x7f21ec0a7f50 2021-03-09T09:26:48.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08dcc0 con 0x7f21ec0a7f50 2021-03-09T09:26:48.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 rx=0x7f21e4017ee0 tx=0x7f21e4017af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:48.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21ec0a7f50 2021-03-09T09:26:48.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21ec0a7f50 2021-03-09T09:26:48.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21ec0a7f50 2021-03-09T09:26:48.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.615+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:48.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:48.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:48.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:50.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:26:50.018 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:26:53.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:53.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:53.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 rx=0x7f21e4017ee0 tx=0x7f21e4017af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:53.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 rx=0x7f21e4017ee0 tx=0x7f21e4017af0).stop 2021-03-09T09:26:53.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:53.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:53.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:53.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:53.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:53.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ea90 con 0x7f21cc005e30 2021-03-09T09:26:53.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d940 con 0x7f21c4005da0 2021-03-09T09:26:53.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ef90 con 0x7f21c40033c0 2021-03-09T09:26:53.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:53.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:53.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:53.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:53.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:53.612 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13594 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:53.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:53.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:53.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:53.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:53.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0153c0 con 0x7f21c40033c0 2021-03-09T09:26:53.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:26:53.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc068a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:53.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b5040 con 0x7f21c40033c0 2021-03-09T09:26:53.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c40033c0 2021-03-09T09:26:53.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc070da0 con 0x7f21c40033c0 2021-03-09T09:26:53.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.615+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:53.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:53.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:53.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:26:58.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:58.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:58.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc068a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:58.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc068a90).stop 2021-03-09T09:26:58.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:58.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=CLOSED pgs=4109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:58.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:58.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:58.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:58.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d940 con 0x7f21cc005e30 2021-03-09T09:26:58.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0153c0 con 0x7f21c4005da0 2021-03-09T09:26:58.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:26:58.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:26:58.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:58.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:26:58.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:58.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:58.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:58.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:58.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:58.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:58.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ef90 con 0x7f21ec0a7f50 2021-03-09T09:26:58.617 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13640 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:26:58.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:26:58.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4120 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc015b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:58.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba090 con 0x7f21ec0a7f50 2021-03-09T09:26:58.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ab220 con 0x7f21ec0a7f50 2021-03-09T09:26:58.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc097520 con 0x7f21ec0a7f50 2021-03-09T09:26:58.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.619+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:26:58.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.619+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:26:58.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:26:58.619+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:03.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:27:03.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:03.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4120 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc015b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:03.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4120 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc015b50).stop 2021-03-09T09:27:03.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:03.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:03.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:03.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:03.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:03.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ef90 con 0x7f21cc005e30 2021-03-09T09:27:03.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0153c0 con 0x7f21c4005da0 2021-03-09T09:27:03.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e00 con 0x7f21c40033c0 2021-03-09T09:27:03.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:03.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:03.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:03.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:03.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:03.618 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:03.611+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13664 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:03.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:03.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:03.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:03.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:03.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0215f0 con 0x7f21c40033c0 2021-03-09T09:27:03.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c40033c0 2021-03-09T09:27:03.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.615+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc028430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:03.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089e40 con 0x7f21c40033c0 2021-03-09T09:27:03.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028820 con 0x7f21c40033c0 2021-03-09T09:27:03.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:27:03.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.623+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:03.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.623+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:03.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:03.623+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:08.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:08.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:08.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc028430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:08.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc028430).stop 2021-03-09T09:27:08.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:08.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093910 unknown :-1 s=CLOSED pgs=4131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:08.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:08.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:08.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:08.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c3d0 con 0x7f21cc005e30 2021-03-09T09:27:08.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089c70 con 0x7f21c4005da0 2021-03-09T09:27:08.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:27:08.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:08.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:08.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:08.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:08.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:08.840 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:08.614+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13709 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:08.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:08.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:08.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:08.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:08.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:27:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08dcc0 con 0x7f21ec0a7f50 2021-03-09T09:27:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e4053f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017e60 con 0x7f21ec0a7f50 2021-03-09T09:27:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007cb0 con 0x7f21ec0a7f50 2021-03-09T09:27:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007cb0 con 0x7f21ec0a7f50 2021-03-09T09:27:08.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.626+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:08.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.626+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:08.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:08.626+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:13.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:13.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:13.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e4053f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:13.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e4053f40).stop 2021-03-09T09:27:13.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:13.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:13.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:13.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:13.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:13.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a0d0 con 0x7f21cc005e30 2021-03-09T09:27:13.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T09:27:13.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c3d0 con 0x7f21c40033c0 2021-03-09T09:27:13.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:13.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:13.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.614+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:13.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:13.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:13.624 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13744 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:13.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:13.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:13.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:13.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:13.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089c70 con 0x7f21c40033c0 2021-03-09T09:27:13.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08dcc0 con 0x7f21c40033c0 2021-03-09T09:27:13.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021780 secure :-1 s=READY pgs=4155 cs=0 l=1 rev1=1 rx=0x7f21cc03f960 tx=0x7f21e40b1190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:13.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21c40033c0 2021-03-09T09:27:13.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c40033c0 2021-03-09T09:27:13.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21c40033c0 2021-03-09T09:27:13.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.626+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:13.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.626+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:13.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:13.626+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:18.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc021780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:18.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc021780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:18.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021780 secure :-1 s=READY pgs=4155 cs=0 l=1 rev1=1 rx=0x7f21cc03f960 tx=0x7f21e40b1190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:18.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021780 secure :-1 s=READY pgs=4155 cs=0 l=1 rev1=1 rx=0x7f21cc03f960 tx=0x7f21e40b1190).stop 2021-03-09T09:27:18.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc021780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:18.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021780 unknown :-1 s=CLOSED pgs=4155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:18.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:18.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:18.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:18.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021e10 con 0x7f21cc005e30 2021-03-09T09:27:18.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T09:27:18.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c570 con 0x7f21ec0a7f50 2021-03-09T09:27:18.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:18.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:18.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:18.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:18.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:18.628 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13784 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:18.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:18.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:18.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:18.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:18.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08a220 con 0x7f21ec0a7f50 2021-03-09T09:27:18.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0215f0 con 0x7f21ec0a7f50 2021-03-09T09:27:18.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 rx=0x7f21cc01f240 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:18.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21ec0a7f50 2021-03-09T09:27:18.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0abeb0 con 0x7f21ec0a7f50 2021-03-09T09:27:18.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc068880 con 0x7f21ec0a7f50 2021-03-09T09:27:18.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.630+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:18.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:18.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:18.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:20.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:27:20.085 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:27:23.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 rx=0x7f21cc01f240 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 rx=0x7f21cc01f240 tx=0x7f21dc009810).stop 2021-03-09T09:27:23.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:23.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:23.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:23.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007970 con 0x7f21cc005e30 2021-03-09T09:27:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec30 con 0x7f21c4005da0 2021-03-09T09:27:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030970 con 0x7f21c40033c0 2021-03-09T09:27:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:23.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:23.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:23.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:23.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:23.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0153c0 con 0x7f21c40033c0 2021-03-09T09:27:23.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08a220 con 0x7f21c40033c0 2021-03-09T09:27:23.632 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:23.622+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13821 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:23.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021460 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 rx=0x7f21dc084010 tx=0x7f21dc003fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:23.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0abeb0 con 0x7f21c40033c0 2021-03-09T09:27:23.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c40033c0 2021-03-09T09:27:23.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07a000 con 0x7f21c40033c0 2021-03-09T09:27:23.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.630+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:23.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:23.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:23.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:28.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:27:28.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:28.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021460 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 rx=0x7f21dc084010 tx=0x7f21dc003fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:28.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021460 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 rx=0x7f21dc084010 tx=0x7f21dc003fb0).stop 2021-03-09T09:27:28.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:28.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc021460 unknown :-1 s=CLOSED pgs=4173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:28.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:28.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:28.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:28.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057830 con 0x7f21cc005e30 2021-03-09T09:27:28.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec30 con 0x7f21c4005da0 2021-03-09T09:27:28.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007970 con 0x7f21ec0a7f50 2021-03-09T09:27:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:28.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:28.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:28.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:28.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0153c0 con 0x7f21ec0a7f50 2021-03-09T09:27:28.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089c70 con 0x7f21ec0a7f50 2021-03-09T09:27:28.635 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:28.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021950 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 rx=0x7f21e40b12e0 tx=0x7f21e4027ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:28.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f3a0 con 0x7f21ec0a7f50 2021-03-09T09:27:28.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f3a0 con 0x7f21ec0a7f50 2021-03-09T09:27:28.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028120 con 0x7f21ec0a7f50 2021-03-09T09:27:28.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.634+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:28.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.634+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:28.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:28.634+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:33.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:33.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:33.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021950 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 rx=0x7f21e40b12e0 tx=0x7f21e4027ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:33.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021950 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 rx=0x7f21e40b12e0 tx=0x7f21e4027ee0).stop 2021-03-09T09:27:33.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc021950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:33.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc021950 unknown :-1 s=CLOSED pgs=4184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:33.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:33.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:33.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:33.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0153c0 con 0x7f21cc005e30 2021-03-09T09:27:33.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T09:27:33.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0215a0 con 0x7f21c40033c0 2021-03-09T09:27:33.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:33.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:33.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:33.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:33.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:33.637 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13891 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:33.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:33.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:33.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:33.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:33.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021710 con 0x7f21c40033c0 2021-03-09T09:27:33.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08a220 con 0x7f21c40033c0 2021-03-09T09:27:33.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 rx=0x7f21cc033f80 tx=0x7f21dc081ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:33.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089e40 con 0x7f21c40033c0 2021-03-09T09:27:33.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089e40 con 0x7f21c40033c0 2021-03-09T09:27:33.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc007c50 con 0x7f21c40033c0 2021-03-09T09:27:33.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.642+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:33.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.646+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:33.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:33.646+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:38.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:27:38.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:38.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 rx=0x7f21cc033f80 tx=0x7f21dc081ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:38.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 rx=0x7f21cc033f80 tx=0x7f21dc081ed0).stop 2021-03-09T09:27:38.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:38.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=CLOSED pgs=4195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:38.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:38.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:38.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:38.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21cc005e30 2021-03-09T09:27:38.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ef90 con 0x7f21c4005da0 2021-03-09T09:27:38.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040070 con 0x7f21ec0a7f50 2021-03-09T09:27:38.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:38.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:38.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:38.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:38.639 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13927 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:38.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:38.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:38.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:38.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:38.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:38.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0401e0 con 0x7f21ec0a7f50 2021-03-09T09:27:38.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021710 con 0x7f21ec0a7f50 2021-03-09T09:27:38.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4211 cs=0 l=1 rev1=1 rx=0x7f21dc06a350 tx=0x7f21dc078550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:38.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:27:38.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:27:38.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:27:38.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.642+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:38.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.642+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:38.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:38.642+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:43.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:43.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:43.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4211 cs=0 l=1 rev1=1 rx=0x7f21dc06a350 tx=0x7f21dc078550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:43.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 secure :-1 s=READY pgs=4211 cs=0 l=1 rev1=1 rx=0x7f21dc06a350 tx=0x7f21dc078550).stop 2021-03-09T09:27:43.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01eaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:43.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01eaa0 unknown :-1 s=CLOSED pgs=4211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:43.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:43.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:43.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:43.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:43.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022670 con 0x7f21cc005e30 2021-03-09T09:27:43.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:43.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040070 con 0x7f21c4005da0 2021-03-09T09:27:43.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0153c0 con 0x7f21c40033c0 2021-03-09T09:27:43.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:43.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:43.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:43.643 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13965 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:43.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:43.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:43.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:43.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:43.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0216a0 con 0x7f21c40033c0 2021-03-09T09:27:43.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0401e0 con 0x7f21c40033c0 2021-03-09T09:27:43.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 rx=0x7f21cc02fcc0 tx=0x7f21dc091b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:43.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099020 con 0x7f21c40033c0 2021-03-09T09:27:43.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007e60 con 0x7f21c40033c0 2021-03-09T09:27:43.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:27:43.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.642+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:43.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.646+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:43.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:43.646+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:48.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:48.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:48.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 rx=0x7f21cc02fcc0 tx=0x7f21dc091b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:48.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 rx=0x7f21cc02fcc0 tx=0x7f21dc091b50).stop 2021-03-09T09:27:48.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:48.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=CLOSED pgs=4216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:48.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:48.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:48.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:48.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045f10 con 0x7f21cc005e30 2021-03-09T09:27:48.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024370 con 0x7f21c4005da0 2021-03-09T09:27:48.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:27:48.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:48.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:48.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:48.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:48.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:48.645 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13999 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:48.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:48.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:48.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:48.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:48.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0401d0 con 0x7f21ec0a7f50 2021-03-09T09:27:48.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.638+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0216a0 con 0x7f21ec0a7f50 2021-03-09T09:27:48.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.642+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fce0 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc031de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:48.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085dc0 con 0x7f21ec0a7f50 2021-03-09T09:27:48.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085dc0 con 0x7f21ec0a7f50 2021-03-09T09:27:48.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.654+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:27:48.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.654+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:48.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.654+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:48.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:48.654+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:50.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:27:50.152 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:27:53.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:53.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:53.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fce0 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc031de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:53.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fce0 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc031de0).stop 2021-03-09T09:27:53.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:53.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fce0 unknown :-1 s=CLOSED pgs=4229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:53.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:53.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:53.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:53.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0567e0 con 0x7f21cc005e30 2021-03-09T09:27:53.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e8a0 con 0x7f21c4005da0 2021-03-09T09:27:53.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dbb0 con 0x7f21c40033c0 2021-03-09T09:27:53.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:53.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:53.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:53.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.637+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:53.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:53.648 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14037 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:53.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:53.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:53.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:53.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:53.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0153c0 con 0x7f21c40033c0 2021-03-09T09:27:53.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0401d0 con 0x7f21c40033c0 2021-03-09T09:27:53.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 rx=0x7f21dc0b6490 tx=0x7f21dc07a970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:53.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00a940 con 0x7f21c40033c0 2021-03-09T09:27:53.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc055e20 con 0x7f21c40033c0 2021-03-09T09:27:53.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07ad80 con 0x7f21c40033c0 2021-03-09T09:27:53.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.649+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:53.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.649+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:53.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:53.649+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:27:58.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:58.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:58.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 rx=0x7f21dc0b6490 tx=0x7f21dc07a970).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:58.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 rx=0x7f21dc0b6490 tx=0x7f21dc07a970).stop 2021-03-09T09:27:58.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:58.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=CLOSED pgs=4239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:58.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:58.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:58.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:58.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009420 con 0x7f21cc005e30 2021-03-09T09:27:58.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dbb0 con 0x7f21c4005da0 2021-03-09T09:27:58.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045f10 con 0x7f21ec0a7f50 2021-03-09T09:27:58.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:27:58.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:27:58.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:58.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:58.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:58.651 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14068 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:27:58.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:58.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:58.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:58.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:58.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:27:58.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0153c0 con 0x7f21ec0a7f50 2021-03-09T09:27:58.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03faa0 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 rx=0x7f21dc0b73e0 tx=0x7f21dc07a900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:58.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21ec0a7f50 2021-03-09T09:27:58.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc004de0 con 0x7f21ec0a7f50 2021-03-09T09:27:58.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:27:58.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.653+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:27:58.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:27:58.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:27:58.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:03.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:03.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:03.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03faa0 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 rx=0x7f21dc0b73e0 tx=0x7f21dc07a900).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:03.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03faa0 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 rx=0x7f21dc0b73e0 tx=0x7f21dc07a900).stop 2021-03-09T09:28:03.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03faa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:03.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03faa0 unknown :-1 s=CLOSED pgs=4249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:03.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:03.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:03.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:03.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009420 con 0x7f21cc005e30 2021-03-09T09:28:03.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045f10 con 0x7f21c4005da0 2021-03-09T09:28:03.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056640 con 0x7f21c40033c0 2021-03-09T09:28:03.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:03.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:03.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:03.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:03.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:03.654 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14106 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:03.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:03.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:03.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:03.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:03.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056780 con 0x7f21c40033c0 2021-03-09T09:28:03.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0153c0 con 0x7f21c40033c0 2021-03-09T09:28:03.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4258 cs=0 l=1 rev1=1 rx=0x7f21dc068390 tx=0x7f21dc00a9b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:03.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099040 con 0x7f21c40033c0 2021-03-09T09:28:03.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21c40033c0 2021-03-09T09:28:03.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:28:03.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.653+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:03.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:03.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:03.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:08.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:08.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:08.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4258 cs=0 l=1 rev1=1 rx=0x7f21dc068390 tx=0x7f21dc00a9b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:08.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4258 cs=0 l=1 rev1=1 rx=0x7f21dc068390 tx=0x7f21dc00a9b0).stop 2021-03-09T09:28:08.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:08.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=CLOSED pgs=4258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:08.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:08.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:08.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:08.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0115b0 con 0x7f21cc005e30 2021-03-09T09:28:08.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009420 con 0x7f21c4005da0 2021-03-09T09:28:08.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:08.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fba0 con 0x7f21ec0a7f50 2021-03-09T09:28:08.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:08.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:08.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:08.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:08.657 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14143 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:08.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:08.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:08.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:08.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:08.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0565f0 con 0x7f21ec0a7f50 2021-03-09T09:28:08.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056780 con 0x7f21ec0a7f50 2021-03-09T09:28:08.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.653+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 rx=0x7f21dc0b6490 tx=0x7f21dc00aa20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:08.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:28:08.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc064000 con 0x7f21ec0a7f50 2021-03-09T09:28:08.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:28:08.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.657+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:08.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.657+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:08.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:08.657+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:13.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:13.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:13.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 rx=0x7f21dc0b6490 tx=0x7f21dc00aa20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:13.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 rx=0x7f21dc0b6490 tx=0x7f21dc00aa20).stop 2021-03-09T09:28:13.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fa50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:13.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 unknown :-1 s=CLOSED pgs=4269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:13.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:13.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:13.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:13.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0866d0 con 0x7f21cc005e30 2021-03-09T09:28:13.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0862c0 con 0x7f21c4005da0 2021-03-09T09:28:13.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0152f0 con 0x7f21c40033c0 2021-03-09T09:28:13.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:13.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:13.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:13.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:13.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.649+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:13.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:13.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:13.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:13.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:13.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d370 con 0x7f21c40033c0 2021-03-09T09:28:13.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089c70 con 0x7f21c40033c0 2021-03-09T09:28:13.663 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14180 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:13.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.653+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 rx=0x7f21e40a51f0 tx=0x7f21e40b1190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:13.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027d20 con 0x7f21c40033c0 2021-03-09T09:28:13.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c40033c0 2021-03-09T09:28:13.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40310c0 con 0x7f21c40033c0 2021-03-09T09:28:13.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.657+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:13.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.661+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:13.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:13.661+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:18.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:18.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:18.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 rx=0x7f21e40a51f0 tx=0x7f21e40b1190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:18.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 rx=0x7f21e40a51f0 tx=0x7f21e40b1190).stop 2021-03-09T09:28:18.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:18.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8d0 unknown :-1 s=CLOSED pgs=4280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:18.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:18.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:18.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:18.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0866d0 con 0x7f21cc005e30 2021-03-09T09:28:18.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011790 con 0x7f21c4005da0 2021-03-09T09:28:18.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0152f0 con 0x7f21ec0a7f50 2021-03-09T09:28:18.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:18.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:18.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:18.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:18.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.653+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:18.663 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14214 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:18.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:18.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:18.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:18.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:18.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d0c0 con 0x7f21ec0a7f50 2021-03-09T09:28:18.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0565f0 con 0x7f21ec0a7f50 2021-03-09T09:28:18.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 rx=0x7f21dc069fd0 tx=0x7f21dc0b6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:18.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b12a0 con 0x7f21ec0a7f50 2021-03-09T09:28:18.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b1070 con 0x7f21ec0a7f50 2021-03-09T09:28:18.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21ec0a7f50 2021-03-09T09:28:18.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.665+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:18.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.665+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:18.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:18.665+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:20.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:28:20.205 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:28:23.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:23.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:23.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 rx=0x7f21dc069fd0 tx=0x7f21dc0b6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:23.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 rx=0x7f21dc069fd0 tx=0x7f21dc0b6190).stop 2021-03-09T09:28:23.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03fa50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:23.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03fa50 unknown :-1 s=CLOSED pgs=4291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:23.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:23.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:23.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:23.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048fc0 con 0x7f21cc005e30 2021-03-09T09:28:23.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012fe0 con 0x7f21c4005da0 2021-03-09T09:28:23.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011790 con 0x7f21c40033c0 2021-03-09T09:28:23.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:23.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:23.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:23.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:23.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:23.667 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14252 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:23.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:23.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:23.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:23.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:23.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0152f0 con 0x7f21c40033c0 2021-03-09T09:28:23.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d0c0 con 0x7f21c40033c0 2021-03-09T09:28:23.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8b0 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 rx=0x7f21dc083660 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:23.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099020 con 0x7f21c40033c0 2021-03-09T09:28:23.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c40033c0 2021-03-09T09:28:23.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21c40033c0 2021-03-09T09:28:23.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.665+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:23.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.665+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:23.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:23.665+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:28.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:28.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:28.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8b0 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 rx=0x7f21dc083660 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:28.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8b0 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 rx=0x7f21dc083660 tx=0x7f21dc010d90).stop 2021-03-09T09:28:28.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04e8b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:28.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04e8b0 unknown :-1 s=CLOSED pgs=4301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:28.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:28.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:28.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:28.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0152f0 con 0x7f21cc005e30 2021-03-09T09:28:28.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:28.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21c4005da0 2021-03-09T09:28:28.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fa50 con 0x7f21ec0a7f50 2021-03-09T09:28:28.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:28.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:28.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:28.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:28.669 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14287 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:28.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:28.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:28.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:28.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:28.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03fb90 con 0x7f21ec0a7f50 2021-03-09T09:28:28.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.661+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d0c0 con 0x7f21ec0a7f50 2021-03-09T09:28:28.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 rx=0x7f21cc0120b0 tx=0x7f21dc0b3200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:28.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099020 con 0x7f21ec0a7f50 2021-03-09T09:28:28.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:28:28.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:28:28.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.669+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:28.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.673+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:28.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:28.673+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:33.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:33.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:33.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 rx=0x7f21cc0120b0 tx=0x7f21dc0b3200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:33.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 rx=0x7f21cc0120b0 tx=0x7f21dc0b3200).stop 2021-03-09T09:28:33.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:33.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=4314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:33.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:33.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:33.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:33.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d0c0 con 0x7f21cc005e30 2021-03-09T09:28:33.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02c700 con 0x7f21c4005da0 2021-03-09T09:28:33.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ab0 con 0x7f21c40033c0 2021-03-09T09:28:33.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:33.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:33.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:33.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:33.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:33.673 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14326 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:33.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:33.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:33.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:33.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:33.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03fd20 con 0x7f21c40033c0 2021-03-09T09:28:33.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03fb90 con 0x7f21c40033c0 2021-03-09T09:28:33.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015260 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 rx=0x7f21dc085dc0 tx=0x7f21dc0b6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:33.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc096ea0 con 0x7f21c40033c0 2021-03-09T09:28:33.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc097070 con 0x7f21c40033c0 2021-03-09T09:28:33.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:28:33.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.673+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:33.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.673+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:33.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:33.673+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:38.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:38.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:38.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015260 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 rx=0x7f21dc085dc0 tx=0x7f21dc0b6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:38.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015260 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 rx=0x7f21dc085dc0 tx=0x7f21dc0b6190).stop 2021-03-09T09:28:38.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:38.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015260 unknown :-1 s=CLOSED pgs=4322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:38.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:38.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:38.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:38.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048aa0 con 0x7f21cc005e30 2021-03-09T09:28:38.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d0c0 con 0x7f21c4005da0 2021-03-09T09:28:38.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ab0 con 0x7f21ec0a7f50 2021-03-09T09:28:38.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:38.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:38.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:38.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:38.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.664+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:38.675 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14361 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:38.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:38.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:38.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:38.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:38.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03fb90 con 0x7f21ec0a7f50 2021-03-09T09:28:38.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03fd20 con 0x7f21ec0a7f50 2021-03-09T09:28:38.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc056a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:38.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21ec0a7f50 2021-03-09T09:28:38.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3310 con 0x7f21ec0a7f50 2021-03-09T09:28:38.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:28:38.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.676+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:38.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.676+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:38.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:38.676+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:43.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:43.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:43.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc056a60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:43.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc056a60).stop 2021-03-09T09:28:43.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:43.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=4333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:43.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:43.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:43.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:43.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048900 con 0x7f21cc005e30 2021-03-09T09:28:43.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048a40 con 0x7f21c4005da0 2021-03-09T09:28:43.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015260 con 0x7f21c40033c0 2021-03-09T09:28:43.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:43.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:43.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:43.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:43.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:43.679 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14402 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:43.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.668+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:43.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:43.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:43.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:43.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02c700 con 0x7f21c40033c0 2021-03-09T09:28:43.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03fb90 con 0x7f21c40033c0 2021-03-09T09:28:43.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 rx=0x7f21dc096bd0 tx=0x7f21dc08c370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:43.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099040 con 0x7f21c40033c0 2021-03-09T09:28:43.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b4e10 con 0x7f21c40033c0 2021-03-09T09:28:43.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:28:43.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.676+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:43.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.680+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:43.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:43.680+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:48.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:28:48.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:48.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 rx=0x7f21dc096bd0 tx=0x7f21dc08c370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:48.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 rx=0x7f21dc096bd0 tx=0x7f21dc08c370).stop 2021-03-09T09:28:48.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:48.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 unknown :-1 s=CLOSED pgs=4345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:48.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:48.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:48.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:48.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03e9c0 con 0x7f21cc005e30 2021-03-09T09:28:48.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fb90 con 0x7f21c4005da0 2021-03-09T09:28:48.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086120 con 0x7f21ec0a7f50 2021-03-09T09:28:48.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:48.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:48.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:48.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:48.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:48.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:48.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:48.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:48.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:48.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc048760 con 0x7f21ec0a7f50 2021-03-09T09:28:48.683 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14438 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:48.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d370 con 0x7f21ec0a7f50 2021-03-09T09:28:48.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 secure :-1 s=READY pgs=4354 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4017190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:48.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21ec0a7f50 2021-03-09T09:28:48.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024160 con 0x7f21ec0a7f50 2021-03-09T09:28:48.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.680+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21ec0a7f50 2021-03-09T09:28:48.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.680+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:48.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.680+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:48.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:48.680+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:50.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:28:50.276 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:28:53.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 secure :-1 s=READY pgs=4354 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4017190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 secure :-1 s=READY pgs=4354 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4017190).stop 2021-03-09T09:28:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 unknown :-1 s=CLOSED pgs=4354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:53.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031af0 con 0x7f21cc005e30 2021-03-09T09:28:53.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048a90 con 0x7f21c4005da0 2021-03-09T09:28:53.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048bd0 con 0x7f21c40033c0 2021-03-09T09:28:53.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:53.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:53.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:53.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:53.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:53.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc048d10 con 0x7f21c40033c0 2021-03-09T09:28:53.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02c700 con 0x7f21c40033c0 2021-03-09T09:28:53.686 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14477 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:53.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc045bd0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07b3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:53.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.680+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21c40033c0 2021-03-09T09:28:53.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.680+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21c40033c0 2021-03-09T09:28:53.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b6200 con 0x7f21c40033c0 2021-03-09T09:28:53.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.684+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:53.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.684+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:53.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:53.684+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:28:58.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:58.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:58.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc045bd0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07b3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:58.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc045bd0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07b3c0).stop 2021-03-09T09:28:58.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:58.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=4365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:58.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:58.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:58.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:58.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03e370 con 0x7f21cc005e30 2021-03-09T09:28:58.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21c4005da0 2021-03-09T09:28:58.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21ec0a7f50 2021-03-09T09:28:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:28:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:28:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:58.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.676+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:58.687 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14512 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:28:58.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03fb90 con 0x7f21ec0a7f50 2021-03-09T09:28:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.680+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc048760 con 0x7f21ec0a7f50 2021-03-09T09:28:58.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 secure :-1 s=READY pgs=4379 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407d0f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:58.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21ec0a7f50 2021-03-09T09:28:58.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024160 con 0x7f21ec0a7f50 2021-03-09T09:28:58.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21ec0a7f50 2021-03-09T09:28:58.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.688+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:28:58.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.692+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:28:58.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:28:58.692+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:03.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:03.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:03.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 secure :-1 s=READY pgs=4379 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407d0f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:03.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 secure :-1 s=READY pgs=4379 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407d0f0).stop 2021-03-09T09:29:03.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:03.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015260 unknown :-1 s=CLOSED pgs=4379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:03.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:03.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:03.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:03.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21cc005e30 2021-03-09T09:29:03.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21c4005da0 2021-03-09T09:29:03.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052c40 con 0x7f21c40033c0 2021-03-09T09:29:03.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:03.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:03.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:03.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:03.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:03.702 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14548 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:03.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:03.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:03.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:03.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:03.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:29:03.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03fb90 con 0x7f21c40033c0 2021-03-09T09:29:03.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 secure :-1 s=READY pgs=4390 cs=0 l=1 rev1=1 rx=0x7f21cc03f9f0 tx=0x7f21e407d390).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:03.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21c40033c0 2021-03-09T09:29:03.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c40033c0 2021-03-09T09:29:03.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21c40033c0 2021-03-09T09:29:03.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.692+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:03.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.696+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:03.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:03.696+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:08.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:08.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:08.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 secure :-1 s=READY pgs=4390 cs=0 l=1 rev1=1 rx=0x7f21cc03f9f0 tx=0x7f21e407d390).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:08.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 secure :-1 s=READY pgs=4390 cs=0 l=1 rev1=1 rx=0x7f21cc03f9f0 tx=0x7f21e407d390).stop 2021-03-09T09:29:08.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:08.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fd20 unknown :-1 s=CLOSED pgs=4390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:08.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:08.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:08.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:08.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21cc005e30 2021-03-09T09:29:08.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21c4005da0 2021-03-09T09:29:08.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015260 con 0x7f21ec0a7f50 2021-03-09T09:29:08.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:08.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:08.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:08.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:08.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:08.695 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14586 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:08.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:08.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:08.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:08.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:08.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0153a0 con 0x7f21ec0a7f50 2021-03-09T09:29:08.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc048d10 con 0x7f21ec0a7f50 2021-03-09T09:29:08.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4397 cs=0 l=1 rev1=1 rx=0x7f21cc050aa0 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:08.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:29:08.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21ec0a7f50 2021-03-09T09:29:08.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc070d30 con 0x7f21ec0a7f50 2021-03-09T09:29:08.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.696+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:08.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.696+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:08.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:08.696+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:13.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:13.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:13.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4397 cs=0 l=1 rev1=1 rx=0x7f21cc050aa0 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:13.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4397 cs=0 l=1 rev1=1 rx=0x7f21cc050aa0 tx=0x7f21dc009880).stop 2021-03-09T09:29:13.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:13.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=4397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:13.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:13.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:13.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:13.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21cc005e30 2021-03-09T09:29:13.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015260 con 0x7f21c4005da0 2021-03-09T09:29:13.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21c40033c0 2021-03-09T09:29:13.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:13.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:13.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:13.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:13.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:13.697 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14622 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:13.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:13.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:13.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:13.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:13.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040040 con 0x7f21c40033c0 2021-03-09T09:29:13.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.688+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:29:13.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fb50 secure :-1 s=READY pgs=4410 cs=0 l=1 rev1=1 rx=0x7f21cc025520 tx=0x7f21e4053ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:13.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404a0d0 con 0x7f21c40033c0 2021-03-09T09:29:13.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404a0d0 con 0x7f21c40033c0 2021-03-09T09:29:13.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4086de0 con 0x7f21c40033c0 2021-03-09T09:29:13.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.696+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:13.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.696+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:13.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:13.696+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:18.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:18.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:18.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fb50 secure :-1 s=READY pgs=4410 cs=0 l=1 rev1=1 rx=0x7f21cc025520 tx=0x7f21e4053ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:18.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fb50 secure :-1 s=READY pgs=4410 cs=0 l=1 rev1=1 rx=0x7f21cc025520 tx=0x7f21e4053ff0).stop 2021-03-09T09:29:18.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03fb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:18.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03fb50 unknown :-1 s=CLOSED pgs=4410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:18.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:18.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:18.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:18.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015260 con 0x7f21cc005e30 2021-03-09T09:29:18.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c9d0 con 0x7f21c4005da0 2021-03-09T09:29:18.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b2f0 con 0x7f21ec0a7f50 2021-03-09T09:29:18.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:18.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:18.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:18.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:18.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:18.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:18.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:18.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:18.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:18.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03e350 con 0x7f21ec0a7f50 2021-03-09T09:29:18.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040040 con 0x7f21ec0a7f50 2021-03-09T09:29:18.702 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14660 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:18.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4418 cs=0 l=1 rev1=1 rx=0x7f21e40b04c0 tx=0x7f21e4003a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:18.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400de20 con 0x7f21ec0a7f50 2021-03-09T09:29:18.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400de20 con 0x7f21ec0a7f50 2021-03-09T09:29:18.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21ec0a7f50 2021-03-09T09:29:18.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.700+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:18.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.704+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:18.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:18.704+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:20.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:29:20.325 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:29:23.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:23.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:23.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4418 cs=0 l=1 rev1=1 rx=0x7f21e40b04c0 tx=0x7f21e4003a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:23.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4418 cs=0 l=1 rev1=1 rx=0x7f21e40b04c0 tx=0x7f21e4003a10).stop 2021-03-09T09:29:23.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:23.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=4418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:23.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:23.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:23.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:23.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21cc005e30 2021-03-09T09:29:23.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21c4005da0 2021-03-09T09:29:23.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015260 con 0x7f21c40033c0 2021-03-09T09:29:23.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:23.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:23.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:23.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:23.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:23.704 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14698 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:23.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:23.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:23.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:23.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:23.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b2f0 con 0x7f21c40033c0 2021-03-09T09:29:23.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0153a0 con 0x7f21c40033c0 2021-03-09T09:29:23.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c790 secure :-1 s=READY pgs=4431 cs=0 l=1 rev1=1 rx=0x7f21dc07b390 tx=0x7f21dc076e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:23.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:29:23.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:29:23.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:29:23.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.703+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:23.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.707+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:23.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:23.707+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:28.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:28.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:28.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c790 secure :-1 s=READY pgs=4431 cs=0 l=1 rev1=1 rx=0x7f21dc07b390 tx=0x7f21dc076e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:28.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c790 secure :-1 s=READY pgs=4431 cs=0 l=1 rev1=1 rx=0x7f21dc07b390 tx=0x7f21dc076e70).stop 2021-03-09T09:29:28.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03c790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:28.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03c790 unknown :-1 s=CLOSED pgs=4431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:28.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:28.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:28.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:28.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21cc005e30 2021-03-09T09:29:28.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21c4005da0 2021-03-09T09:29:28.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ce20 con 0x7f21ec0a7f50 2021-03-09T09:29:28.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:28.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:28.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:28.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:28.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:28.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:28.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:28.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:28.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:28.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c9d0 con 0x7f21ec0a7f50 2021-03-09T09:29:28.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03e350 con 0x7f21ec0a7f50 2021-03-09T09:29:28.709 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:28.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4440 cs=0 l=1 rev1=1 rx=0x7f21cc052510 tx=0x7f21e4017400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:28.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5a90 con 0x7f21ec0a7f50 2021-03-09T09:29:28.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4092e80 con 0x7f21ec0a7f50 2021-03-09T09:29:28.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21ec0a7f50 2021-03-09T09:29:28.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.707+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:28.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.707+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:28.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:28.707+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:33.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:33.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:33.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4440 cs=0 l=1 rev1=1 rx=0x7f21cc052510 tx=0x7f21e4017400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:33.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 secure :-1 s=READY pgs=4440 cs=0 l=1 rev1=1 rx=0x7f21cc052510 tx=0x7f21e4017400).stop 2021-03-09T09:29:33.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:33.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=4440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:33.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:33.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:33.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:33.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b70 con 0x7f21cc005e30 2021-03-09T09:29:33.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096920 con 0x7f21c4005da0 2021-03-09T09:29:33.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21c40033c0 2021-03-09T09:29:33.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:33.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:33.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:33.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:33.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:33.711 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14772 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:33.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:33.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:33.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:33.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:33.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:29:33.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c9d0 con 0x7f21c40033c0 2021-03-09T09:29:33.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.707+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f020 secure :-1 s=READY pgs=4450 cs=0 l=1 rev1=1 rx=0x7f21e4096ff0 tx=0x7f21e402d480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:33.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40489d0 con 0x7f21c40033c0 2021-03-09T09:29:33.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1500 con 0x7f21c40033c0 2021-03-09T09:29:33.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21c40033c0 2021-03-09T09:29:33.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.711+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:33.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.715+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:33.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:33.715+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:38.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:38.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:38.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f020 secure :-1 s=READY pgs=4450 cs=0 l=1 rev1=1 rx=0x7f21e4096ff0 tx=0x7f21e402d480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:38.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f020 secure :-1 s=READY pgs=4450 cs=0 l=1 rev1=1 rx=0x7f21e4096ff0 tx=0x7f21e402d480).stop 2021-03-09T09:29:38.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:38.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=4450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:38.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:38.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:38.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:38.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21cc005e30 2021-03-09T09:29:38.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d2a0 con 0x7f21c4005da0 2021-03-09T09:29:38.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d030 con 0x7f21ec0a7f50 2021-03-09T09:29:38.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:38.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:38.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:38.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:38.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:38.716 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14809 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:38.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:38.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:38.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:38.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:38.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03c8f0 con 0x7f21ec0a7f50 2021-03-09T09:29:38.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b2f0 con 0x7f21ec0a7f50 2021-03-09T09:29:38.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.711+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c400 secure :-1 s=READY pgs=4462 cs=0 l=1 rev1=1 rx=0x7f21cc030990 tx=0x7f21dc020480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:38.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc096810 con 0x7f21ec0a7f50 2021-03-09T09:29:38.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007390 con 0x7f21ec0a7f50 2021-03-09T09:29:38.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052dc0 con 0x7f21ec0a7f50 2021-03-09T09:29:38.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.719+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:38.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.719+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:38.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:38.719+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:43.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:43.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:43.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c400 secure :-1 s=READY pgs=4462 cs=0 l=1 rev1=1 rx=0x7f21cc030990 tx=0x7f21dc020480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:43.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c400 secure :-1 s=READY pgs=4462 cs=0 l=1 rev1=1 rx=0x7f21cc030990 tx=0x7f21dc020480).stop 2021-03-09T09:29:43.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03c400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:43.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03c400 unknown :-1 s=CLOSED pgs=4462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:43.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:43.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:43.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:43.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052e10 con 0x7f21cc005e30 2021-03-09T09:29:43.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015230 con 0x7f21c4005da0 2021-03-09T09:29:43.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045bd0 con 0x7f21c40033c0 2021-03-09T09:29:43.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:43.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:43.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:43.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:43.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.707+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:43.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:43.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:43.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:43.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:43.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc045d40 con 0x7f21c40033c0 2021-03-09T09:29:43.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:29:43.719 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14844 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:43.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.711+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052920 secure :-1 s=READY pgs=4472 cs=0 l=1 rev1=1 rx=0x7f21cc0916f0 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:43.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21c40033c0 2021-03-09T09:29:43.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4036d30 con 0x7f21c40033c0 2021-03-09T09:29:43.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21c40033c0 2021-03-09T09:29:43.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.715+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:43.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.719+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:43.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:43.719+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:48.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc052920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:48.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc052920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:48.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052920 secure :-1 s=READY pgs=4472 cs=0 l=1 rev1=1 rx=0x7f21cc0916f0 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:48.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052920 secure :-1 s=READY pgs=4472 cs=0 l=1 rev1=1 rx=0x7f21cc0916f0 tx=0x7f21e4053ed0).stop 2021-03-09T09:29:48.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc052920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:48.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052920 unknown :-1 s=CLOSED pgs=4472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:48.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:48.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:48.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:48.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:29:48.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041060 con 0x7f21c4005da0 2021-03-09T09:29:48.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045bd0 con 0x7f21ec0a7f50 2021-03-09T09:29:48.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:48.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:48.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:48.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:48.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.711+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:48.720 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14883 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:48.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:48.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:48.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:48.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:48.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03e350 con 0x7f21ec0a7f50 2021-03-09T09:29:48.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03c8f0 con 0x7f21ec0a7f50 2021-03-09T09:29:48.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01b120 secure :-1 s=READY pgs=4485 cs=0 l=1 rev1=1 rx=0x7f21dc005cb0 tx=0x7f21dc0b5f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:48.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21ec0a7f50 2021-03-09T09:29:48.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07b110 con 0x7f21ec0a7f50 2021-03-09T09:29:48.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052dc0 con 0x7f21ec0a7f50 2021-03-09T09:29:48.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.723+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:48.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.723+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:48.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:48.723+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:50.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:29:50.401 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:29:53.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:53.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:53.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01b120 secure :-1 s=READY pgs=4485 cs=0 l=1 rev1=1 rx=0x7f21dc005cb0 tx=0x7f21dc0b5f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:53.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01b120 secure :-1 s=READY pgs=4485 cs=0 l=1 rev1=1 rx=0x7f21dc005cb0 tx=0x7f21dc0b5f10).stop 2021-03-09T09:29:53.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01b120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:53.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01b120 unknown :-1 s=CLOSED pgs=4485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:53.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:53.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:53.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:53.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:29:53.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d720 con 0x7f21c4005da0 2021-03-09T09:29:53.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045bd0 con 0x7f21c40033c0 2021-03-09T09:29:53.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:53.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:53.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:53.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:53.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:53.722 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:53.715+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14915 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:53.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:53.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:53.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:53.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:53.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0411b0 con 0x7f21c40033c0 2021-03-09T09:29:53.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:29:53.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052900 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc08ce30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:53.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21c40033c0 2021-03-09T09:29:53.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052dc0 con 0x7f21c40033c0 2021-03-09T09:29:53.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052dc0 con 0x7f21c40033c0 2021-03-09T09:29:53.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.723+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:53.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.727+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:53.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:53.727+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:29:58.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc052900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:58.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc052900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:58.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052900 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc08ce30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:58.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052900 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc08ce30).stop 2021-03-09T09:29:58.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc052900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:58.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc052900 unknown :-1 s=CLOSED pgs=4494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:58.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:58.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:58.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:58.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:29:58.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092710 con 0x7f21c4005da0 2021-03-09T09:29:58.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045bd0 con 0x7f21ec0a7f50 2021-03-09T09:29:58.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:29:58.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:58.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:29:58.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:58.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:58.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:58.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:58.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:58.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:58.727 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14956 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:29:58.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03e350 con 0x7f21ec0a7f50 2021-03-09T09:29:58.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0411b0 con 0x7f21ec0a7f50 2021-03-09T09:29:58.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4504 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:58.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21ec0a7f50 2021-03-09T09:29:58.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc064d80 con 0x7f21ec0a7f50 2021-03-09T09:29:58.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.727+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:29:58.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.727+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:29:58.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.727+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:29:58.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:29:58.727+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:03.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:03.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:03.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4504 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:03.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4504 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc010d90).stop 2021-03-09T09:30:03.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:03.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=4504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:03.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:03.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:03.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:03.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:03.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e650 con 0x7f21c4005da0 2021-03-09T09:30:03.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fcc0 con 0x7f21c40033c0 2021-03-09T09:30:03.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:03.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:03.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:03.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.719+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:03.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:03.729 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14991 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:03.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:03.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:03.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:03.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:03.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012310 con 0x7f21c40033c0 2021-03-09T09:30:03.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:30:03.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.727+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ac40 secure :-1 s=READY pgs=4516 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc064ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:03.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.727+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21c40033c0 2021-03-09T09:30:03.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.727+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0691d0 con 0x7f21c40033c0 2021-03-09T09:30:03.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.735+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:30:03.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.735+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:03.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.735+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:03.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:03.735+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:08.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:08.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:08.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ac40 secure :-1 s=READY pgs=4516 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc064ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:08.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ac40 secure :-1 s=READY pgs=4516 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc064ef0).stop 2021-03-09T09:30:08.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ac40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:08.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ac40 unknown :-1 s=CLOSED pgs=4516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:08.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:08.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:08.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:08.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093cd0 con 0x7f21cc005e30 2021-03-09T09:30:08.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21c4005da0 2021-03-09T09:30:08.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ae0 con 0x7f21ec0a7f50 2021-03-09T09:30:08.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:08.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:08.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:08.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.723+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:08.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:08.733 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15027 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:08.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:08.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:08.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:08.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:08.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03e350 con 0x7f21ec0a7f50 2021-03-09T09:30:08.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc045d40 con 0x7f21ec0a7f50 2021-03-09T09:30:08.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.727+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4525 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e40a5120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:08.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.731+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:30:08.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.731+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:30:08.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.735+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4048180 con 0x7f21ec0a7f50 2021-03-09T09:30:08.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.735+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:08.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.735+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:08.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:08.735+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:13.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:13.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:13.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4525 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e40a5120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:13.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4525 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e40a5120).stop 2021-03-09T09:30:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=4525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:13.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022b50 con 0x7f21c4005da0 2021-03-09T09:30:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093cd0 con 0x7f21c40033c0 2021-03-09T09:30:13.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:13.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:13.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:13.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:13.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:13.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:13.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:30:13.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03e350 con 0x7f21c40033c0 2021-03-09T09:30:13.737 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:13.726+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15065 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:13.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f510 secure :-1 s=READY pgs=4536 cs=0 l=1 rev1=1 rx=0x7f21e407d2b0 tx=0x7f21e4017210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:13.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c40033c0 2021-03-09T09:30:13.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21c40033c0 2021-03-09T09:30:13.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c40033c0 2021-03-09T09:30:13.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.738+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:13.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.738+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:13.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:13.738+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:18.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:30:18.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:18.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f510 secure :-1 s=READY pgs=4536 cs=0 l=1 rev1=1 rx=0x7f21e407d2b0 tx=0x7f21e4017210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:18.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f510 secure :-1 s=READY pgs=4536 cs=0 l=1 rev1=1 rx=0x7f21e407d2b0 tx=0x7f21e4017210).stop 2021-03-09T09:30:18.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc04f510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:18.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc04f510 unknown :-1 s=CLOSED pgs=4536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:18.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:18.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:18.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:18.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:18.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022b50 con 0x7f21c4005da0 2021-03-09T09:30:18.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ae0 con 0x7f21ec0a7f50 2021-03-09T09:30:18.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:18.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:18.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:18.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:18.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:18.738 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15103 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:18.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:18.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:18.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:18.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:18.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093cd0 con 0x7f21ec0a7f50 2021-03-09T09:30:18.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012310 con 0x7f21ec0a7f50 2021-03-09T09:30:18.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4547 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:18.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068bb0 con 0x7f21ec0a7f50 2021-03-09T09:30:18.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc055880 con 0x7f21ec0a7f50 2021-03-09T09:30:18.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055d80 con 0x7f21ec0a7f50 2021-03-09T09:30:18.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.738+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:18.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.738+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:18.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:18.738+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:20.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:30:20.466 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:30:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4547 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4547 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009810).stop 2021-03-09T09:30:23.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:23.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=4547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:23.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:23.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:23.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:23.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:23.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ae0 con 0x7f21c4005da0 2021-03-09T09:30:23.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015200 con 0x7f21c40033c0 2021-03-09T09:30:23.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:23.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:23.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:23.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:23.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:23.742 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:23.734+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15138 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:23.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:23.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:23.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:23.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:23.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054b20 con 0x7f21c40033c0 2021-03-09T09:30:23.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093cd0 con 0x7f21c40033c0 2021-03-09T09:30:23.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 secure :-1 s=READY pgs=4563 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc068ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:23.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21c40033c0 2021-03-09T09:30:23.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028e20 con 0x7f21c40033c0 2021-03-09T09:30:23.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21c40033c0 2021-03-09T09:30:23.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.746+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:23.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:23.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:23.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:28.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc054630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:28.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc054630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:28.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 secure :-1 s=READY pgs=4563 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc068ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:28.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 secure :-1 s=READY pgs=4563 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc068ca0).stop 2021-03-09T09:30:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc054630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 unknown :-1 s=CLOSED pgs=4563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022cf0 con 0x7f21c4005da0 2021-03-09T09:30:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ae0 con 0x7f21ec0a7f50 2021-03-09T09:30:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:28.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:28.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:28.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:28.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:28.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:28.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093cd0 con 0x7f21ec0a7f50 2021-03-09T09:30:28.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054b20 con 0x7f21ec0a7f50 2021-03-09T09:30:28.746 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15177 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:28.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4568 cs=0 l=1 rev1=1 rx=0x7f21dc02dea0 tx=0x7f21dc083380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:28.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21ec0a7f50 2021-03-09T09:30:28.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056ce0 con 0x7f21ec0a7f50 2021-03-09T09:30:28.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b1910 con 0x7f21ec0a7f50 2021-03-09T09:30:28.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.742+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:28.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:28.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:28.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:33.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:33.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:33.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4568 cs=0 l=1 rev1=1 rx=0x7f21dc02dea0 tx=0x7f21dc083380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:33.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 secure :-1 s=READY pgs=4568 cs=0 l=1 rev1=1 rx=0x7f21dc02dea0 tx=0x7f21dc083380).stop 2021-03-09T09:30:33.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:33.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04f020 unknown :-1 s=CLOSED pgs=4568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:33.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:33.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:33.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:33.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:33.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ae0 con 0x7f21c4005da0 2021-03-09T09:30:33.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21c40033c0 2021-03-09T09:30:33.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:33.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:33.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:33.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:33.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.738+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:33.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:33.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:33.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:33.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:33.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015200 con 0x7f21c40033c0 2021-03-09T09:30:33.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:30:33.749 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15212 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:33.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 secure :-1 s=READY pgs=4578 cs=0 l=1 rev1=1 rx=0x7f21cc048cb0 tx=0x7f21e407d1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:33.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21c40033c0 2021-03-09T09:30:33.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4008320 con 0x7f21c40033c0 2021-03-09T09:30:33.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21c40033c0 2021-03-09T09:30:33.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.750+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:33.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.750+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:33.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:33.750+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:38.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc054630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:38.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc054630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:38.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 secure :-1 s=READY pgs=4578 cs=0 l=1 rev1=1 rx=0x7f21cc048cb0 tx=0x7f21e407d1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:38.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 secure :-1 s=READY pgs=4578 cs=0 l=1 rev1=1 rx=0x7f21cc048cb0 tx=0x7f21e407d1d0).stop 2021-03-09T09:30:38.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc054630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:38.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc054630 unknown :-1 s=CLOSED pgs=4578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:38.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:38.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:38.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:38.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:38.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08ea10 con 0x7f21c4005da0 2021-03-09T09:30:38.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21ec0a7f50 2021-03-09T09:30:38.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:38.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:38.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:38.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:38.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:38.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:38.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:38.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:38.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:38.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21ec0a7f50 2021-03-09T09:30:38.753 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15251 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:38.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093cd0 con 0x7f21ec0a7f50 2021-03-09T09:30:38.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4590 cs=0 l=1 rev1=1 rx=0x7f21dc028c20 tx=0x7f21dc064ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:38.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc056ce0 con 0x7f21ec0a7f50 2021-03-09T09:30:38.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028c50 con 0x7f21ec0a7f50 2021-03-09T09:30:38.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:30:38.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.754+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:38.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:38.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:38.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:43.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:43.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:43.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4590 cs=0 l=1 rev1=1 rx=0x7f21dc028c20 tx=0x7f21dc064ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:43.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4590 cs=0 l=1 rev1=1 rx=0x7f21dc028c20 tx=0x7f21dc064ef0).stop 2021-03-09T09:30:43.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:43.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=CLOSED pgs=4590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:43.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:43.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:43.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:43.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:43.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0984f0 con 0x7f21c4005da0 2021-03-09T09:30:43.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21c40033c0 2021-03-09T09:30:43.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:43.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:43.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:43.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:43.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:43.754 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:43.746+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15286 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:43.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:43.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:43.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:43.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:43.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054ce0 con 0x7f21c40033c0 2021-03-09T09:30:43.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:30:43.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4604 cs=0 l=1 rev1=1 rx=0x7f21e4041760 tx=0x7f21e407d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:43.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b0eb0 con 0x7f21c40033c0 2021-03-09T09:30:43.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0eb0 con 0x7f21c40033c0 2021-03-09T09:30:43.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21c40033c0 2021-03-09T09:30:43.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.758+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:43.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.758+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:43.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:43.758+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:48.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:48.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:48.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4604 cs=0 l=1 rev1=1 rx=0x7f21e4041760 tx=0x7f21e407d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:48.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4604 cs=0 l=1 rev1=1 rx=0x7f21e4041760 tx=0x7f21e407d160).stop 2021-03-09T09:30:48.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:48.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=CLOSED pgs=4604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:48.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:48.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:48.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:48.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:48.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052aa0 con 0x7f21c4005da0 2021-03-09T09:30:48.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0198e0 con 0x7f21ec0a7f50 2021-03-09T09:30:48.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:48.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:48.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:48.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:48.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:48.758 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15324 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:48.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:48.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:48.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:48.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:48.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093cd0 con 0x7f21ec0a7f50 2021-03-09T09:30:48.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21ec0a7f50 2021-03-09T09:30:48.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4610 cs=0 l=1 rev1=1 rx=0x7f21cc025550 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:48.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0707f0 con 0x7f21ec0a7f50 2021-03-09T09:30:48.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028c50 con 0x7f21ec0a7f50 2021-03-09T09:30:48.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:30:48.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.758+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:48.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.758+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:48.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:48.758+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:50.503 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:30:50.506 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:30:53.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:53.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:53.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4610 cs=0 l=1 rev1=1 rx=0x7f21cc025550 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:53.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4610 cs=0 l=1 rev1=1 rx=0x7f21cc025550 tx=0x7f21dc077e50).stop 2021-03-09T09:30:53.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:53.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=CLOSED pgs=4610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:53.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:53.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:53.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:53.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:53.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030e10 con 0x7f21c4005da0 2021-03-09T09:30:53.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21c40033c0 2021-03-09T09:30:53.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:53.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:53.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:53.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:53.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:53.760 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15359 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:53.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:53.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:53.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:53.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:53.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054990 con 0x7f21c40033c0 2021-03-09T09:30:53.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093cd0 con 0x7f21c40033c0 2021-03-09T09:30:53.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4621 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc083230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:53.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b4ef0 con 0x7f21c40033c0 2021-03-09T09:30:53.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc055650 con 0x7f21c40033c0 2021-03-09T09:30:53.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21c40033c0 2021-03-09T09:30:53.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.762+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:53.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:53.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:53.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:30:58.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:58.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:58.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4621 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc083230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:58.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4621 cs=0 l=1 rev1=1 rx=0x7f21dc071060 tx=0x7f21dc083230).stop 2021-03-09T09:30:58.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:58.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=CLOSED pgs=4621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:58.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:58.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:58.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:58.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:30:58.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08da60 con 0x7f21c4005da0 2021-03-09T09:30:58.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024c70 con 0x7f21ec0a7f50 2021-03-09T09:30:58.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:30:58.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:58.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:30:58.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:58.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.754+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:58.763 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15397 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:30:58.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:58.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:58.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:58.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:58.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:30:58.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054990 con 0x7f21ec0a7f50 2021-03-09T09:30:58.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.758+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 rx=0x7f21cc01aef0 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:58.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b4ef0 con 0x7f21ec0a7f50 2021-03-09T09:30:58.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b5e80 con 0x7f21ec0a7f50 2021-03-09T09:30:58.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.766+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00a130 con 0x7f21ec0a7f50 2021-03-09T09:30:58.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.766+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:30:58.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.766+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:30:58.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:30:58.766+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:03.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:03.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:03.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 rx=0x7f21cc01aef0 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:03.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 rx=0x7f21cc01aef0 tx=0x7f21dc009810).stop 2021-03-09T09:31:03.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc04fa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:03.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc04fa70 unknown :-1 s=CLOSED pgs=4632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:03.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:03.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:03.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:03.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:31:03.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056330 con 0x7f21c4005da0 2021-03-09T09:31:03.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054150 con 0x7f21c40033c0 2021-03-09T09:31:03.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:03.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:03.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:03.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:03.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.757+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:03.769 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:03.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:03.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:03.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:03.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:03.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21c40033c0 2021-03-09T09:31:03.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054ce0 con 0x7f21c40033c0 2021-03-09T09:31:03.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4643 cs=0 l=1 rev1=1 rx=0x7f21cc057a00 tx=0x7f21e4048180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:03.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003ad0 con 0x7f21c40033c0 2021-03-09T09:31:03.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21c40033c0 2021-03-09T09:31:03.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21c40033c0 2021-03-09T09:31:03.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.769+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:03.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.769+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:03.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:03.769+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:08.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:08.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:08.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4643 cs=0 l=1 rev1=1 rx=0x7f21cc057a00 tx=0x7f21e4048180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:08.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4643 cs=0 l=1 rev1=1 rx=0x7f21cc057a00 tx=0x7f21e4048180).stop 2021-03-09T09:31:08.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:08.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=CLOSED pgs=4643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:08.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:08.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:08.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:08.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:31:08.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054150 con 0x7f21c4005da0 2021-03-09T09:31:08.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e820 con 0x7f21ec0a7f50 2021-03-09T09:31:08.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:08.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:08.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:08.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:08.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:08.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:08.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:08.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:08.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:08.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093cd0 con 0x7f21ec0a7f50 2021-03-09T09:31:08.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21ec0a7f50 2021-03-09T09:31:08.772 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:08.761+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15467 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:08.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027ca0 secure :-1 s=READY pgs=4653 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e40a5d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:08.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21ec0a7f50 2021-03-09T09:31:08.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f160 con 0x7f21ec0a7f50 2021-03-09T09:31:08.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:31:08.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.769+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:08.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.773+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:08.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:08.773+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:13.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc027ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:13.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc027ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:13.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027ca0 secure :-1 s=READY pgs=4653 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e40a5d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:13.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027ca0 secure :-1 s=READY pgs=4653 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e40a5d00).stop 2021-03-09T09:31:13.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc027ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:13.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027ca0 unknown :-1 s=CLOSED pgs=4653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:13.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:13.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:13.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:13.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:31:13.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00bc10 con 0x7f21c4005da0 2021-03-09T09:31:13.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054150 con 0x7f21c40033c0 2021-03-09T09:31:13.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:13.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:13.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:13.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:13.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:13.774 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15506 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:13.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:13.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:13.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:13.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:13.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054ce0 con 0x7f21c40033c0 2021-03-09T09:31:13.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:31:13.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4665 cs=0 l=1 rev1=1 rx=0x7f21dc028ff0 tx=0x7f21dc0034c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:13.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc055470 con 0x7f21c40033c0 2021-03-09T09:31:13.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0832c0 con 0x7f21c40033c0 2021-03-09T09:31:13.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc028d50 con 0x7f21c40033c0 2021-03-09T09:31:13.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.773+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:13.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:13.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:13.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:18.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:18.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:18.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4665 cs=0 l=1 rev1=1 rx=0x7f21dc028ff0 tx=0x7f21dc0034c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:18.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4665 cs=0 l=1 rev1=1 rx=0x7f21dc028ff0 tx=0x7f21dc0034c0).stop 2021-03-09T09:31:18.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:18.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=CLOSED pgs=4665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:18.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:18.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:18.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:18.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021f70 con 0x7f21cc005e30 2021-03-09T09:31:18.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030dd0 con 0x7f21c4005da0 2021-03-09T09:31:18.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf80 con 0x7f21ec0a7f50 2021-03-09T09:31:18.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:18.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:18.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:18.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:18.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:18.777 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15544 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:18.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:18.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:18.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:18.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:18.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022d10 con 0x7f21ec0a7f50 2021-03-09T09:31:18.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054ce0 con 0x7f21ec0a7f50 2021-03-09T09:31:18.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027960 secure :-1 s=READY pgs=4675 cs=0 l=1 rev1=1 rx=0x7f21dc0b1210 tx=0x7f21dc018240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:18.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21ec0a7f50 2021-03-09T09:31:18.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0832a0 con 0x7f21ec0a7f50 2021-03-09T09:31:18.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:31:18.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.777+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:18.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:18.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:18.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:20.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:31:20.554 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:31:23.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc027960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:23.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc027960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:23.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027960 secure :-1 s=READY pgs=4675 cs=0 l=1 rev1=1 rx=0x7f21dc0b1210 tx=0x7f21dc018240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:23.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027960 secure :-1 s=READY pgs=4675 cs=0 l=1 rev1=1 rx=0x7f21dc0b1210 tx=0x7f21dc018240).stop 2021-03-09T09:31:23.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc027960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:23.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc027960 unknown :-1 s=CLOSED pgs=4675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:23.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:23.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:23.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:23.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21cc005e30 2021-03-09T09:31:23.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017350 con 0x7f21c4005da0 2021-03-09T09:31:23.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00bc30 con 0x7f21c40033c0 2021-03-09T09:31:23.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:23.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:23.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:23.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:23.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:23.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:23.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:23.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:23.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:23.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21c40033c0 2021-03-09T09:31:23.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093cd0 con 0x7f21c40033c0 2021-03-09T09:31:23.782 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15577 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:23.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4685 cs=0 l=1 rev1=1 rx=0x7f21e4096040 tx=0x7f21e4036e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:23.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c40033c0 2021-03-09T09:31:23.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0e80 con 0x7f21c40033c0 2021-03-09T09:31:23.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c40033c0 2021-03-09T09:31:23.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.781+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:23.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.781+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:23.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:23.781+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:28.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:28.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:28.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4685 cs=0 l=1 rev1=1 rx=0x7f21e4096040 tx=0x7f21e4036e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:28.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 secure :-1 s=READY pgs=4685 cs=0 l=1 rev1=1 rx=0x7f21e4096040 tx=0x7f21e4036e60).stop 2021-03-09T09:31:28.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:28.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015200 unknown :-1 s=CLOSED pgs=4685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:28.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:28.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0278c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:28.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:28.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ea00 con 0x7f21cc005e30 2021-03-09T09:31:28.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237c0 con 0x7f21c4005da0 2021-03-09T09:31:28.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0866d0 con 0x7f21ec0a7f50 2021-03-09T09:31:28.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:28.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:28.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:28.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:28.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:28.783 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:28.773+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15617 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:28.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:28.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:28.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:28.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:28.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054ce0 con 0x7f21ec0a7f50 2021-03-09T09:31:28.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21ec0a7f50 2021-03-09T09:31:28.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0278c0 secure :-1 s=READY pgs=4696 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e40b0e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:28.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024c90 con 0x7f21ec0a7f50 2021-03-09T09:31:28.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21ec0a7f50 2021-03-09T09:31:28.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21ec0a7f50 2021-03-09T09:31:28.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.785+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:28.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.785+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:28.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:28.785+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:33.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0278c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:33.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0278c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:33.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0278c0 secure :-1 s=READY pgs=4696 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e40b0e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:33.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0278c0 secure :-1 s=READY pgs=4696 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e40b0e80).stop 2021-03-09T09:31:33.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0278c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:33.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0278c0 unknown :-1 s=CLOSED pgs=4696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:33.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:33.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:33.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:33.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089990 con 0x7f21cc005e30 2021-03-09T09:31:33.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21c4005da0 2021-03-09T09:31:33.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ab0 con 0x7f21c40033c0 2021-03-09T09:31:33.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:33.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:33.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:33.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:33.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:33.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:33.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:33.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:33.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:33.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00bc10 con 0x7f21c40033c0 2021-03-09T09:31:33.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022d10 con 0x7f21c40033c0 2021-03-09T09:31:33.788 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:33.777+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15653 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:33.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151d0 secure :-1 s=READY pgs=4706 cs=0 l=1 rev1=1 rx=0x7f21cc02c8c0 tx=0x7f21dc096e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:33.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21c40033c0 2021-03-09T09:31:33.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21c40033c0 2021-03-09T09:31:33.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:31:33.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.789+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:33.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.789+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:33.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:33.789+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:38.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:38.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:38.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151d0 secure :-1 s=READY pgs=4706 cs=0 l=1 rev1=1 rx=0x7f21cc02c8c0 tx=0x7f21dc096e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:38.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151d0 secure :-1 s=READY pgs=4706 cs=0 l=1 rev1=1 rx=0x7f21cc02c8c0 tx=0x7f21dc096e70).stop 2021-03-09T09:31:38.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0151d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:38.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151d0 unknown :-1 s=CLOSED pgs=4706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:38.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:38.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:38.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:38.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc059110 con 0x7f21cc005e30 2021-03-09T09:31:38.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237c0 con 0x7f21c4005da0 2021-03-09T09:31:38.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21ec0a7f50 2021-03-09T09:31:38.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:38.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:38.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:38.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:38.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:38.789 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15690 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:38.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:38.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:38.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:38.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:38.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21ec0a7f50 2021-03-09T09:31:38.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00bc10 con 0x7f21ec0a7f50 2021-03-09T09:31:38.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 secure :-1 s=READY pgs=4717 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:38.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21ec0a7f50 2021-03-09T09:31:38.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21ec0a7f50 2021-03-09T09:31:38.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc003800 con 0x7f21ec0a7f50 2021-03-09T09:31:38.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.789+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:38.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.789+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:38.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:38.789+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:43.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:43.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:43.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 secure :-1 s=READY pgs=4717 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:43.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 secure :-1 s=READY pgs=4717 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc009880).stop 2021-03-09T09:31:43.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02f940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:43.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 unknown :-1 s=CLOSED pgs=4717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:43.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:43.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:43.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:43.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec20 con 0x7f21cc005e30 2021-03-09T09:31:43.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056650 con 0x7f21c4005da0 2021-03-09T09:31:43.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21c40033c0 2021-03-09T09:31:43.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:43.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:43.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:43.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:43.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:43.792 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15726 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:43.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:43.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:43.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:43.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:43.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03ebb0 con 0x7f21c40033c0 2021-03-09T09:31:43.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21c40033c0 2021-03-09T09:31:43.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151a0 secure :-1 s=READY pgs=4728 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc0280e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:43.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21c40033c0 2021-03-09T09:31:43.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06d000 con 0x7f21c40033c0 2021-03-09T09:31:43.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c40033c0 2021-03-09T09:31:43.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.789+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:43.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:43.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:43.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:48.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:48.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:48.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151a0 secure :-1 s=READY pgs=4728 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc0280e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:48.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151a0 secure :-1 s=READY pgs=4728 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc0280e0).stop 2021-03-09T09:31:48.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0151a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:48.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0151a0 unknown :-1 s=CLOSED pgs=4728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:48.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:48.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:48.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:48.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21cc005e30 2021-03-09T09:31:48.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ab0 con 0x7f21c4005da0 2021-03-09T09:31:48.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00bc10 con 0x7f21ec0a7f50 2021-03-09T09:31:48.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:48.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:48.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:48.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:48.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:48.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:48.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:48.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:48.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:48.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02fe30 con 0x7f21ec0a7f50 2021-03-09T09:31:48.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03ebb0 con 0x7f21ec0a7f50 2021-03-09T09:31:48.796 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:48.785+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15763 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:48.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 secure :-1 s=READY pgs=4738 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc0034c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:48.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00afa0 con 0x7f21ec0a7f50 2021-03-09T09:31:48.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00afa0 con 0x7f21ec0a7f50 2021-03-09T09:31:48.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00a130 con 0x7f21ec0a7f50 2021-03-09T09:31:48.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.793+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:48.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:48.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:48.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:50.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:31:50.601 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:31:53.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:53.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 secure :-1 s=READY pgs=4738 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc0034c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 secure :-1 s=READY pgs=4738 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc0034c0).stop 2021-03-09T09:31:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02f940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02f940 unknown :-1 s=CLOSED pgs=4738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21cc005e30 2021-03-09T09:31:53.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03cc30 con 0x7f21c4005da0 2021-03-09T09:31:53.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ab0 con 0x7f21c40033c0 2021-03-09T09:31:53.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:53.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:53.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:53.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:53.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:53.798 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15799 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:53.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:53.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:53.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:53.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:53.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024c30 con 0x7f21c40033c0 2021-03-09T09:31:53.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03ebb0 con 0x7f21c40033c0 2021-03-09T09:31:53.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015100 secure :-1 s=READY pgs=4751 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc096f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:53.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21c40033c0 2021-03-09T09:31:53.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00afa0 con 0x7f21c40033c0 2021-03-09T09:31:53.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc050900 con 0x7f21c40033c0 2021-03-09T09:31:53.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.800+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:53.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.800+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:53.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:53.800+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:31:58.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:58.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:58.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015100 secure :-1 s=READY pgs=4751 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc096f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:58.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015100 secure :-1 s=READY pgs=4751 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc096f90).stop 2021-03-09T09:31:58.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc015100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:58.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc015100 unknown :-1 s=CLOSED pgs=4751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:58.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0155f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:58.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:58.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:58.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae10 con 0x7f21cc005e30 2021-03-09T09:31:58.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c4005da0 2021-03-09T09:31:58.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ae0 con 0x7f21ec0a7f50 2021-03-09T09:31:58.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:31:58.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:58.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:31:58.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:58.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0155f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:58.801 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15836 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:31:58.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:58.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:58.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:58.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:58.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00bc10 con 0x7f21ec0a7f50 2021-03-09T09:31:58.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.792+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024c30 con 0x7f21ec0a7f50 2021-03-09T09:31:58.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0155f0 secure :-1 s=READY pgs=4762 cs=0 l=1 rev1=1 rx=0x7f21dc0035a0 tx=0x7f21dc083150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:58.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b7c30 con 0x7f21ec0a7f50 2021-03-09T09:31:58.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00afa0 con 0x7f21ec0a7f50 2021-03-09T09:31:58.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc050900 con 0x7f21ec0a7f50 2021-03-09T09:31:58.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.800+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:31:58.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.800+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:31:58.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:31:58.800+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:03.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0155f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:03.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0155f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:03.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0155f0 secure :-1 s=READY pgs=4762 cs=0 l=1 rev1=1 rx=0x7f21dc0035a0 tx=0x7f21dc083150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:03.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0155f0 secure :-1 s=READY pgs=4762 cs=0 l=1 rev1=1 rx=0x7f21dc0035a0 tx=0x7f21dc083150).stop 2021-03-09T09:32:03.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0155f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:03.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0155f0 unknown :-1 s=CLOSED pgs=4762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:03.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:03.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc02f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:03.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:03.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024c30 con 0x7f21cc005e30 2021-03-09T09:32:03.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c4005da0 2021-03-09T09:32:03.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21c40033c0 2021-03-09T09:32:03.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc02f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:03.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:03.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:03.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:03.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:03.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:03.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:03.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:03.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:03.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02fe30 con 0x7f21c40033c0 2021-03-09T09:32:03.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054ce0 con 0x7f21c40033c0 2021-03-09T09:32:03.806 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15872 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:03.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc02f940 secure :-1 s=READY pgs=4771 cs=0 l=1 rev1=1 rx=0x7f21e4007ba0 tx=0x7f21e401af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:03.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21c40033c0 2021-03-09T09:32:03.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007c10 con 0x7f21c40033c0 2021-03-09T09:32:03.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007c10 con 0x7f21c40033c0 2021-03-09T09:32:03.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.800+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:03.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.804+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:03.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:03.804+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:08.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc02f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:08.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc02f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:08.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc02f940 secure :-1 s=READY pgs=4771 cs=0 l=1 rev1=1 rx=0x7f21e4007ba0 tx=0x7f21e401af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:08.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc02f940 secure :-1 s=READY pgs=4771 cs=0 l=1 rev1=1 rx=0x7f21e4007ba0 tx=0x7f21e401af80).stop 2021-03-09T09:32:08.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc02f940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:08.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc02f940 unknown :-1 s=CLOSED pgs=4771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:08.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:08.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:08.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:08.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022400 con 0x7f21cc005e30 2021-03-09T09:32:08.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f140 con 0x7f21c4005da0 2021-03-09T09:32:08.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21ec0a7f50 2021-03-09T09:32:08.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:08.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:08.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:08.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:08.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:08.807 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15909 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:08.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:08.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:08.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:08.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:08.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024c30 con 0x7f21ec0a7f50 2021-03-09T09:32:08.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02fe30 con 0x7f21ec0a7f50 2021-03-09T09:32:08.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014dc0 secure :-1 s=READY pgs=4781 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e4028d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:08.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a3280 con 0x7f21ec0a7f50 2021-03-09T09:32:08.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3400 con 0x7f21ec0a7f50 2021-03-09T09:32:08.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21ec0a7f50 2021-03-09T09:32:08.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.804+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:08.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.808+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:08.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:08.808+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:13.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc014dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:13.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc014dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:13.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014dc0 secure :-1 s=READY pgs=4781 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e4028d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:13.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014dc0 secure :-1 s=READY pgs=4781 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e4028d60).stop 2021-03-09T09:32:13.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc014dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:13.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc014dc0 unknown :-1 s=CLOSED pgs=4781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:13.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0152b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:13.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:13.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:13.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c2f0 con 0x7f21cc005e30 2021-03-09T09:32:13.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f140 con 0x7f21c4005da0 2021-03-09T09:32:13.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21c40033c0 2021-03-09T09:32:13.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:13.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:13.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:13.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:13.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:13.810 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:13.800+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15942 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:13.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:13.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:13.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:13.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:13.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:32:13.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00bc10 con 0x7f21c40033c0 2021-03-09T09:32:13.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0152b0 secure :-1 s=READY pgs=4797 cs=0 l=1 rev1=1 rx=0x7f21dc033d70 tx=0x7f21dc050f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:13.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21c40033c0 2021-03-09T09:32:13.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0204a0 con 0x7f21c40033c0 2021-03-09T09:32:13.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c40033c0 2021-03-09T09:32:13.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.812+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:13.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.812+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:13.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:13.812+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:18.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:18.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:18.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0152b0 secure :-1 s=READY pgs=4797 cs=0 l=1 rev1=1 rx=0x7f21dc033d70 tx=0x7f21dc050f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:18.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0152b0 secure :-1 s=READY pgs=4797 cs=0 l=1 rev1=1 rx=0x7f21dc033d70 tx=0x7f21dc050f70).stop 2021-03-09T09:32:18.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc0152b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:18.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc0152b0 unknown :-1 s=CLOSED pgs=4797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:18.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:18.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:18.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0157a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:18.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0510d0 con 0x7f21cc005e30 2021-03-09T09:32:18.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21c4005da0 2021-03-09T09:32:18.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b190 con 0x7f21ec0a7f50 2021-03-09T09:32:18.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:18.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:18.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:18.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:18.818 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15983 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:18.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0157a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:18.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:18.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:18.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:18.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:18.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015c90 con 0x7f21ec0a7f50 2021-03-09T09:32:18.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:32:18.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0157a0 secure :-1 s=READY pgs=4806 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc020510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:18.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21ec0a7f50 2021-03-09T09:32:18.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056ab0 con 0x7f21ec0a7f50 2021-03-09T09:32:18.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:32:18.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.816+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:18.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.816+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:18.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:18.816+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:20.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:32:20.652 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:32:23.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0157a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:23.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0157a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:23.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0157a0 secure :-1 s=READY pgs=4806 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc020510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:23.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0157a0 secure :-1 s=READY pgs=4806 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc020510).stop 2021-03-09T09:32:23.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc0157a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:23.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc0157a0 unknown :-1 s=CLOSED pgs=4806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:23.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:23.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:23.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc094040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:23.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b190 con 0x7f21cc005e30 2021-03-09T09:32:23.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c4005da0 2021-03-09T09:32:23.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21c40033c0 2021-03-09T09:32:23.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:23.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:23.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:23.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:23.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc094040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:23.817 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16019 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:23.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:23.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:23.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:23.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:23.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc014a10 con 0x7f21c40033c0 2021-03-09T09:32:23.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015c90 con 0x7f21c40033c0 2021-03-09T09:32:23.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc094040 secure :-1 s=READY pgs=4816 cs=0 l=1 rev1=1 rx=0x7f21cc00e7e0 tx=0x7f21dc096000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:23.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0897b0 con 0x7f21c40033c0 2021-03-09T09:32:23.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21c40033c0 2021-03-09T09:32:23.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:32:23.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.820+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:23.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.820+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:23.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:23.820+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:28.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc094040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:28.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc094040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:28.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc094040 secure :-1 s=READY pgs=4816 cs=0 l=1 rev1=1 rx=0x7f21cc00e7e0 tx=0x7f21dc096000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:28.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc094040 secure :-1 s=READY pgs=4816 cs=0 l=1 rev1=1 rx=0x7f21cc00e7e0 tx=0x7f21dc096000).stop 2021-03-09T09:32:28.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc094040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:28.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc094040 unknown :-1 s=CLOSED pgs=4816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:28.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:28.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:28.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:28.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0512c0 con 0x7f21cc005e30 2021-03-09T09:32:28.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b190 con 0x7f21c4005da0 2021-03-09T09:32:28.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:28.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21ec0a7f50 2021-03-09T09:32:28.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:28.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:28.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:28.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:28.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:28.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:28.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:28.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:28.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:32:28.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024c30 con 0x7f21ec0a7f50 2021-03-09T09:32:28.821 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16057 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:28.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4825 cs=0 l=1 rev1=1 rx=0x7f21e40b7040 tx=0x7f21e40a6f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:28.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007e20 con 0x7f21ec0a7f50 2021-03-09T09:32:28.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007e20 con 0x7f21ec0a7f50 2021-03-09T09:32:28.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:32:28.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.820+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:28.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.820+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:28.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:28.820+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:33.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:33.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:33.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4825 cs=0 l=1 rev1=1 rx=0x7f21e40b7040 tx=0x7f21e40a6f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:33.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4825 cs=0 l=1 rev1=1 rx=0x7f21e40b7040 tx=0x7f21e40a6f40).stop 2021-03-09T09:32:33.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:33.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=CLOSED pgs=4825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:33.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:33.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:33.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:33.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0512c0 con 0x7f21cc005e30 2021-03-09T09:32:33.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024c30 con 0x7f21c4005da0 2021-03-09T09:32:33.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21c40033c0 2021-03-09T09:32:33.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:33.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:33.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:33.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:33.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:33.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:33.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:33.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:33.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:33.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b0f0 con 0x7f21c40033c0 2021-03-09T09:32:33.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc014a10 con 0x7f21c40033c0 2021-03-09T09:32:33.825 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16093 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:33.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093f90 secure :-1 s=READY pgs=4834 cs=0 l=1 rev1=1 rx=0x7f21cc08d810 tx=0x7f21dc0abf80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:33.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01f000 con 0x7f21c40033c0 2021-03-09T09:32:33.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6120 con 0x7f21c40033c0 2021-03-09T09:32:33.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21c40033c0 2021-03-09T09:32:33.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.824+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:33.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:33.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:33.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:38.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:38.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:38.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093f90 secure :-1 s=READY pgs=4834 cs=0 l=1 rev1=1 rx=0x7f21cc08d810 tx=0x7f21dc0abf80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:38.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093f90 secure :-1 s=READY pgs=4834 cs=0 l=1 rev1=1 rx=0x7f21cc08d810 tx=0x7f21dc0abf80).stop 2021-03-09T09:32:38.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc093f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:38.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc093f90 unknown :-1 s=CLOSED pgs=4834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:38.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:38.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:38.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:38.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01afb0 con 0x7f21cc005e30 2021-03-09T09:32:38.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094480 con 0x7f21c4005da0 2021-03-09T09:32:38.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046360 con 0x7f21ec0a7f50 2021-03-09T09:32:38.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:38.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:38.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:38.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:38.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:38.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:38.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:38.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:38.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:38.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024c30 con 0x7f21ec0a7f50 2021-03-09T09:32:38.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b0f0 con 0x7f21ec0a7f50 2021-03-09T09:32:38.829 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16131 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:38.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4845 cs=0 l=1 rev1=1 rx=0x7f21cc056af0 tx=0x7f21dc0766b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:38.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc076210 con 0x7f21ec0a7f50 2021-03-09T09:32:38.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0857f0 con 0x7f21ec0a7f50 2021-03-09T09:32:38.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc076000 con 0x7f21ec0a7f50 2021-03-09T09:32:38.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.832+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:38.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.832+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:38.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:38.832+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:43.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:43.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:43.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4845 cs=0 l=1 rev1=1 rx=0x7f21cc056af0 tx=0x7f21dc0766b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:43.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4845 cs=0 l=1 rev1=1 rx=0x7f21cc056af0 tx=0x7f21dc0766b0).stop 2021-03-09T09:32:43.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:43.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=CLOSED pgs=4845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:43.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:43.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:43.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:43.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c1b0 con 0x7f21cc005e30 2021-03-09T09:32:43.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046500 con 0x7f21c4005da0 2021-03-09T09:32:43.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0148e0 con 0x7f21c40033c0 2021-03-09T09:32:43.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:43.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:43.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:43.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:43.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:43.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:43.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:43.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:43.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:43.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093f20 con 0x7f21c40033c0 2021-03-09T09:32:43.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:32:43.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40a6df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:43.834 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:43.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c40033c0 2021-03-09T09:32:43.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21c40033c0 2021-03-09T09:32:43.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.827+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21c40033c0 2021-03-09T09:32:43.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.827+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:43.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.827+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:43.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:43.827+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:48.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:48.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:48.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40a6df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:48.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 secure :-1 s=READY pgs=4856 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40a6df0).stop 2021-03-09T09:32:48.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:48.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 unknown :-1 s=CLOSED pgs=4856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:48.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:48.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:48.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:48.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b6f0 con 0x7f21cc005e30 2021-03-09T09:32:48.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c1b0 con 0x7f21c4005da0 2021-03-09T09:32:48.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0466a0 con 0x7f21ec0a7f50 2021-03-09T09:32:48.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:48.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:48.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:48.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:48.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:48.834 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16205 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:48.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:48.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:48.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:48.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:48.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:32:48.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024c30 con 0x7f21ec0a7f50 2021-03-09T09:32:48.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.827+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 rx=0x7f21dc081c20 tx=0x7f21dc076470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:48.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.827+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062ee0 con 0x7f21ec0a7f50 2021-03-09T09:32:48.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062ee0 con 0x7f21ec0a7f50 2021-03-09T09:32:48.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc076000 con 0x7f21ec0a7f50 2021-03-09T09:32:48.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.835+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:48.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.835+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:48.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:48.835+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:50.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:32:50.705 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:32:53.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.827+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:53.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.827+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:53.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.827+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 rx=0x7f21dc081c20 tx=0x7f21dc076470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:53.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.827+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 rx=0x7f21dc081c20 tx=0x7f21dc076470).stop 2021-03-09T09:32:53.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.827+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:53.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.827+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=CLOSED pgs=4869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:53.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:53.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:53.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:53.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089cd0 con 0x7f21cc005e30 2021-03-09T09:32:53.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c910 con 0x7f21c4005da0 2021-03-09T09:32:53.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024dd0 con 0x7f21c40033c0 2021-03-09T09:32:53.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:53.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:53.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:53.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:53.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:53.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:53.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:53.838 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16241 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:53.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:53.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:53.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b840 con 0x7f21c40033c0 2021-03-09T09:32:53.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.831+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093f20 con 0x7f21c40033c0 2021-03-09T09:32:53.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 secure :-1 s=READY pgs=4880 cs=0 l=1 rev1=1 rx=0x7f21e4092060 tx=0x7f21e4027bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:53.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21c40033c0 2021-03-09T09:32:53.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40312c0 con 0x7f21c40033c0 2021-03-09T09:32:53.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3070 con 0x7f21c40033c0 2021-03-09T09:32:53.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.843+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:53.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.843+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:53.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:53.843+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:32:58.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:58.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:58.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 secure :-1 s=READY pgs=4880 cs=0 l=1 rev1=1 rx=0x7f21e4092060 tx=0x7f21e4027bd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:58.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 secure :-1 s=READY pgs=4880 cs=0 l=1 rev1=1 rx=0x7f21e4092060 tx=0x7f21e4027bd0).stop 2021-03-09T09:32:58.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:58.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ae10 unknown :-1 s=CLOSED pgs=4880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:58.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:58.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:58.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:58.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c910 con 0x7f21cc005e30 2021-03-09T09:32:58.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052aa0 con 0x7f21c4005da0 2021-03-09T09:32:58.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089cd0 con 0x7f21ec0a7f50 2021-03-09T09:32:58.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:32:58.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:58.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:32:58.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:58.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:58.841 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16279 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:32:58.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:58.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:58.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:58.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:58.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc024dd0 con 0x7f21ec0a7f50 2021-03-09T09:32:58.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b840 con 0x7f21ec0a7f50 2021-03-09T09:32:58.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4889 cs=0 l=1 rev1=1 rx=0x7f21cc01f030 tx=0x7f21e40b1120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:58.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027e00 con 0x7f21ec0a7f50 2021-03-09T09:32:58.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e00 con 0x7f21ec0a7f50 2021-03-09T09:32:58.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3280 con 0x7f21ec0a7f50 2021-03-09T09:32:58.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.843+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:32:58.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.847+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:32:58.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:32:58.847+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:03.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:03.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:03.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4889 cs=0 l=1 rev1=1 rx=0x7f21cc01f030 tx=0x7f21e40b1120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:03.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 secure :-1 s=READY pgs=4889 cs=0 l=1 rev1=1 rx=0x7f21cc01f030 tx=0x7f21e40b1120).stop 2021-03-09T09:33:03.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc015210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:03.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc015210 unknown :-1 s=CLOSED pgs=4889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:03.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:03.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:03.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:03.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023630 con 0x7f21cc005e30 2021-03-09T09:33:03.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0146b0 con 0x7f21c4005da0 2021-03-09T09:33:03.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089990 con 0x7f21c40033c0 2021-03-09T09:33:03.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:03.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:03.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:03.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:03.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.835+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:03.844 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16314 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:03.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:03.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:03.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:03.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:03.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0940c0 con 0x7f21c40033c0 2021-03-09T09:33:03.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:33:03.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ade0 secure :-1 s=READY pgs=4899 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc06dc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:03.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081a60 con 0x7f21c40033c0 2021-03-09T09:33:03.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0764e0 con 0x7f21c40033c0 2021-03-09T09:33:03.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06ddc0 con 0x7f21c40033c0 2021-03-09T09:33:03.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.847+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:03.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.847+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:03.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:03.847+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:08.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ade0 secure :-1 s=READY pgs=4899 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc06dc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:08.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ade0 secure :-1 s=READY pgs=4899 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc06dc90).stop 2021-03-09T09:33:08.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc01ade0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:08.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc01ade0 unknown :-1 s=CLOSED pgs=4899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:08.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:08.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:08.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:08.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0402a0 con 0x7f21cc005e30 2021-03-09T09:33:08.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c4005da0 2021-03-09T09:33:08.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:08.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275c0 con 0x7f21ec0a7f50 2021-03-09T09:33:08.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:08.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:08.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:08.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.839+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:08.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:08.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:08.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:08.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:08.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015210 con 0x7f21ec0a7f50 2021-03-09T09:33:08.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc024dd0 con 0x7f21ec0a7f50 2021-03-09T09:33:08.850 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16353 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:08.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4909 cs=0 l=1 rev1=1 rx=0x7f21cc023900 tx=0x7f21e4016f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:08.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5300 con 0x7f21ec0a7f50 2021-03-09T09:33:08.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21ec0a7f50 2021-03-09T09:33:08.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409ca30 con 0x7f21ec0a7f50 2021-03-09T09:33:08.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.847+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:08.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.851+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:08.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:08.851+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:13.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:13.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:13.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4909 cs=0 l=1 rev1=1 rx=0x7f21cc023900 tx=0x7f21e4016f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:13.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4909 cs=0 l=1 rev1=1 rx=0x7f21cc023900 tx=0x7f21e4016f50).stop 2021-03-09T09:33:13.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:13.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=CLOSED pgs=4909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:13.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:13.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:13.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:13.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0124e0 con 0x7f21cc005e30 2021-03-09T09:33:13.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275c0 con 0x7f21c4005da0 2021-03-09T09:33:13.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:33:13.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:13.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:13.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:13.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:13.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:13.851 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:13.843+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16388 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:13.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:13.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:13.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:13.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:13.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0146b0 con 0x7f21c40033c0 2021-03-09T09:33:13.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015210 con 0x7f21c40033c0 2021-03-09T09:33:13.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.847+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b490 secure :-1 s=READY pgs=4922 cs=0 l=1 rev1=1 rx=0x7f21cc0864d0 tx=0x7f21e403a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:13.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5300 con 0x7f21c40033c0 2021-03-09T09:33:13.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21c40033c0 2021-03-09T09:33:13.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40413f0 con 0x7f21c40033c0 2021-03-09T09:33:13.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.855+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:13.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.855+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:13.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:13.855+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:18.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:18.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:18.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b490 secure :-1 s=READY pgs=4922 cs=0 l=1 rev1=1 rx=0x7f21cc0864d0 tx=0x7f21e403a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:18.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b490 secure :-1 s=READY pgs=4922 cs=0 l=1 rev1=1 rx=0x7f21cc0864d0 tx=0x7f21e403a000).stop 2021-03-09T09:33:18.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03b490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:18.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b490 unknown :-1 s=CLOSED pgs=4922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:18.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:18.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:18.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:18.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275c0 con 0x7f21cc005e30 2021-03-09T09:33:18.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03cec0 con 0x7f21c4005da0 2021-03-09T09:33:18.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f500 con 0x7f21ec0a7f50 2021-03-09T09:33:18.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:18.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:18.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:18.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:18.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:18.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:18.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:18.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:18.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:18.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:33:18.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0940c0 con 0x7f21ec0a7f50 2021-03-09T09:33:18.856 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:18.847+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16427 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:18.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4930 cs=0 l=1 rev1=1 rx=0x7f21cc015160 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:18.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc070000 con 0x7f21ec0a7f50 2021-03-09T09:33:18.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc070000 con 0x7f21ec0a7f50 2021-03-09T09:33:18.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc076000 con 0x7f21ec0a7f50 2021-03-09T09:33:18.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.859+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:18.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.859+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:18.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:18.859+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:20.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:33:20.753 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:33:23.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:23.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.847+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:23.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.847+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4930 cs=0 l=1 rev1=1 rx=0x7f21cc015160 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:23.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.847+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4930 cs=0 l=1 rev1=1 rx=0x7f21cc015160 tx=0x7f21dc009810).stop 2021-03-09T09:33:23.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:23.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.847+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=CLOSED pgs=4930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:23.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:23.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:23.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:23.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275c0 con 0x7f21cc005e30 2021-03-09T09:33:23.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f500 con 0x7f21c4005da0 2021-03-09T09:33:23.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094260 con 0x7f21c40033c0 2021-03-09T09:33:23.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:23.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:23.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:23.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:23.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:23.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:23.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:23.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:23.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:23.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0146b0 con 0x7f21c40033c0 2021-03-09T09:33:23.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21c40033c0 2021-03-09T09:33:23.859 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16462 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:23.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b3f0 secure :-1 s=READY pgs=4941 cs=0 l=1 rev1=1 rx=0x7f21cc03e900 tx=0x7f21dc055070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:23.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21c40033c0 2021-03-09T09:33:23.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc068af0 con 0x7f21c40033c0 2021-03-09T09:33:23.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c40033c0 2021-03-09T09:33:23.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.859+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:23.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.859+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:23.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:23.859+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:28.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:28.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:28.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b3f0 secure :-1 s=READY pgs=4941 cs=0 l=1 rev1=1 rx=0x7f21cc03e900 tx=0x7f21dc055070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:28.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b3f0 secure :-1 s=READY pgs=4941 cs=0 l=1 rev1=1 rx=0x7f21cc03e900 tx=0x7f21dc055070).stop 2021-03-09T09:33:28.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 msgr2=0x7f21cc03b3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c40033c0 0x7f21cc03b3f0 unknown :-1 s=CLOSED pgs=4941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030c80 con 0x7f21cc005e30 2021-03-09T09:33:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275c0 con 0x7f21c4005da0 2021-03-09T09:33:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030610 con 0x7f21ec0a7f50 2021-03-09T09:33:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:28.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:28.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:28.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:28.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:28.865 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16499 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:28.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:28.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:28.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:28.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:28.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:33:28.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0146b0 con 0x7f21ec0a7f50 2021-03-09T09:33:28.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4956 cs=0 l=1 rev1=1 rx=0x7f21dc055a70 tx=0x7f21dc076070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:28.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc050670 con 0x7f21ec0a7f50 2021-03-09T09:33:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050490 con 0x7f21ec0a7f50 2021-03-09T09:33:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc050da0 con 0x7f21ec0a7f50 2021-03-09T09:33:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.867+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.867+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:28.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:28.867+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:33.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:33.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:33.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4956 cs=0 l=1 rev1=1 rx=0x7f21dc055a70 tx=0x7f21dc076070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:33.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 secure :-1 s=READY pgs=4956 cs=0 l=1 rev1=1 rx=0x7f21dc055a70 tx=0x7f21dc076070).stop 2021-03-09T09:33:33.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:33.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089990 unknown :-1 s=CLOSED pgs=4956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:33.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:33.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275c0 con 0x7f21cc005e30 2021-03-09T09:33:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f500 con 0x7f21c4005da0 2021-03-09T09:33:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030c80 con 0x7f21cc0153a0 2021-03-09T09:33:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:33.864 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16536 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:33.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:33.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03b7d0 con 0x7f21cc0153a0 2021-03-09T09:33:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21cc0153a0 2021-03-09T09:33:33.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=4963 cs=0 l=1 rev1=1 rx=0x7f21dc0035a0 tx=0x7f21dc032c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:33.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc050490 con 0x7f21cc0153a0 2021-03-09T09:33:33.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0153a0 2021-03-09T09:33:33.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0153a0 2021-03-09T09:33:33.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.866+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:33.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.866+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:33.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:33.866+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:38.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:38.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:38.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=4963 cs=0 l=1 rev1=1 rx=0x7f21dc0035a0 tx=0x7f21dc032c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:38.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=4963 cs=0 l=1 rev1=1 rx=0x7f21dc0035a0 tx=0x7f21dc032c30).stop 2021-03-09T09:33:38.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:38.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=CLOSED pgs=4963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:38.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:38.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:38.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:38.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b960 con 0x7f21cc005e30 2021-03-09T09:33:38.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015e10 con 0x7f21c4005da0 2021-03-09T09:33:38.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0452e0 con 0x7f21ec0a7f50 2021-03-09T09:33:38.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:38.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:38.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:38.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:38.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.858+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:38.868 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16575 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:38.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:38.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:38.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:38.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:38.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040050 con 0x7f21ec0a7f50 2021-03-09T09:33:38.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03b7d0 con 0x7f21ec0a7f50 2021-03-09T09:33:38.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=4973 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc028f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:38.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b4e60 con 0x7f21ec0a7f50 2021-03-09T09:33:38.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0508a0 con 0x7f21ec0a7f50 2021-03-09T09:33:38.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0553c0 con 0x7f21ec0a7f50 2021-03-09T09:33:38.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.866+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:38.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.870+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:38.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:38.870+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:43.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:43.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:43.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=4973 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc028f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:43.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=4973 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc028f90).stop 2021-03-09T09:33:43.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:43.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=4973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:43.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:43.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:43.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:43.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014750 con 0x7f21cc005e30 2021-03-09T09:33:43.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054f10 con 0x7f21c4005da0 2021-03-09T09:33:43.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0452e0 con 0x7f21cc0153a0 2021-03-09T09:33:43.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:43.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:43.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:43.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:43.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:43.870 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:43.862+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16610 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:43.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:43.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:43.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:43.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:43.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21cc0153a0 2021-03-09T09:33:43.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015210 con 0x7f21cc0153a0 2021-03-09T09:33:43.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089990 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407fdc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:43.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003ad0 con 0x7f21cc0153a0 2021-03-09T09:33:43.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4003ad0 con 0x7f21cc0153a0 2021-03-09T09:33:43.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc0153a0 2021-03-09T09:33:43.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.874+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:43.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.874+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:43.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:43.874+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:48.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:48.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc089990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:48.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089990 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407fdc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:48.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089990 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407fdc0).stop 2021-03-09T09:33:48.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc089990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:48.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089990 unknown :-1 s=CLOSED pgs=4988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:48.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:48.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:48.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:48.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015210 con 0x7f21cc005e30 2021-03-09T09:33:48.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027570 con 0x7f21c4005da0 2021-03-09T09:33:48.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093f20 con 0x7f21ec0a7f50 2021-03-09T09:33:48.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:48.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:48.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:48.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:48.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:48.874 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16649 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:48.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:48.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:48.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:48.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:48.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094090 con 0x7f21ec0a7f50 2021-03-09T09:33:48.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.866+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:33:48.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=4994 cs=0 l=1 rev1=1 rx=0x7f21e4017510 tx=0x7f21e40a5120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:48.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21ec0a7f50 2021-03-09T09:33:48.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40310b0 con 0x7f21ec0a7f50 2021-03-09T09:33:48.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21ec0a7f50 2021-03-09T09:33:48.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.874+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:48.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.874+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:48.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:48.874+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:50.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:33:50.824 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:33:53.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:53.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:53.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=4994 cs=0 l=1 rev1=1 rx=0x7f21e4017510 tx=0x7f21e40a5120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:53.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=4994 cs=0 l=1 rev1=1 rx=0x7f21e4017510 tx=0x7f21e40a5120).stop 2021-03-09T09:33:53.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:53.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=4994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:53.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:53.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:53.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:53.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093f20 con 0x7f21cc005e30 2021-03-09T09:33:53.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c730 con 0x7f21c4005da0 2021-03-09T09:33:53.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21cc0153a0 2021-03-09T09:33:53.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:53.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:53.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:53.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:53.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:53.877 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16684 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:53.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:53.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:53.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:53.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:53.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0145b0 con 0x7f21cc0153a0 2021-03-09T09:33:53.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040050 con 0x7f21cc0153a0 2021-03-09T09:33:53.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 secure :-1 s=READY pgs=5005 cs=0 l=1 rev1=1 rx=0x7f21cc0084d0 tx=0x7f21dc02e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:53.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0153a0 2021-03-09T09:33:53.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09a2a0 con 0x7f21cc0153a0 2021-03-09T09:33:53.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc028ca0 con 0x7f21cc0153a0 2021-03-09T09:33:53.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.878+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:53.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.882+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:53.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:53.882+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:33:58.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:58.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:58.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 secure :-1 s=READY pgs=5005 cs=0 l=1 rev1=1 rx=0x7f21cc0084d0 tx=0x7f21dc02e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:58.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 secure :-1 s=READY pgs=5005 cs=0 l=1 rev1=1 rx=0x7f21cc0084d0 tx=0x7f21dc02e000).stop 2021-03-09T09:33:58.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:58.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 unknown :-1 s=CLOSED pgs=5005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:58.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:58.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:58.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:58.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089de0 con 0x7f21cc005e30 2021-03-09T09:33:58.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093f20 con 0x7f21c4005da0 2021-03-09T09:33:58.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:33:58.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:33:58.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:58.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:33:58.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:58.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:58.880 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16723 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:33:58.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:58.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:58.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:58.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:58.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0145b0 con 0x7f21ec0a7f50 2021-03-09T09:33:58.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094090 con 0x7f21ec0a7f50 2021-03-09T09:33:58.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.878+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 rx=0x7f21cc009a50 tx=0x7f21e40171d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:58.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21ec0a7f50 2021-03-09T09:33:58.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21ec0a7f50 2021-03-09T09:33:58.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21ec0a7f50 2021-03-09T09:33:58.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.886+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:33:58.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.886+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:33:58.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:33:58.886+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:03.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:03.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:03.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 rx=0x7f21cc009a50 tx=0x7f21e40171d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:03.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 rx=0x7f21cc009a50 tx=0x7f21e40171d0).stop 2021-03-09T09:34:03.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:03.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:03.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:03.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:03.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:03.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040480 con 0x7f21cc005e30 2021-03-09T09:34:03.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21c4005da0 2021-03-09T09:34:03.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f5d0 con 0x7f21cc0153a0 2021-03-09T09:34:03.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:03.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:03.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:03.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.874+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:03.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:03.884 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16758 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:03.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:03.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:03.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:03.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:03.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041350 con 0x7f21cc0153a0 2021-03-09T09:34:03.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040050 con 0x7f21cc0153a0 2021-03-09T09:34:03.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 secure :-1 s=READY pgs=5030 cs=0 l=1 rev1=1 rx=0x7f21cc025330 tx=0x7f21dc010e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:03.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089880 con 0x7f21cc0153a0 2021-03-09T09:34:03.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089880 con 0x7f21cc0153a0 2021-03-09T09:34:03.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052da0 con 0x7f21cc0153a0 2021-03-09T09:34:03.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.890+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:03.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.890+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:03.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:03.890+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:08.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:08.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:08.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 secure :-1 s=READY pgs=5030 cs=0 l=1 rev1=1 rx=0x7f21cc025330 tx=0x7f21dc010e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:08.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 secure :-1 s=READY pgs=5030 cs=0 l=1 rev1=1 rx=0x7f21cc025330 tx=0x7f21dc010e00).stop 2021-03-09T09:34:08.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:08.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898f0 unknown :-1 s=CLOSED pgs=5030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:08.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:08.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:08.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:08.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21cc005e30 2021-03-09T09:34:08.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f5d0 con 0x7f21c4005da0 2021-03-09T09:34:08.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027570 con 0x7f21ec0a7f50 2021-03-09T09:34:08.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:08.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:08.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:08.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:08.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:08.889 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16795 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:08.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:08.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:08.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:08.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:08.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03c6e0 con 0x7f21ec0a7f50 2021-03-09T09:34:08.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041350 con 0x7f21ec0a7f50 2021-03-09T09:34:08.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5042 cs=0 l=1 rev1=1 rx=0x7f21cc0461e0 tx=0x7f21dc01f6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:08.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21ec0a7f50 2021-03-09T09:34:08.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc068e50 con 0x7f21ec0a7f50 2021-03-09T09:34:08.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc09a0d0 con 0x7f21ec0a7f50 2021-03-09T09:34:08.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.890+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:08.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.890+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:08.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:08.890+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:13.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:13.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:13.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5042 cs=0 l=1 rev1=1 rx=0x7f21cc0461e0 tx=0x7f21dc01f6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:13.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5042 cs=0 l=1 rev1=1 rx=0x7f21cc0461e0 tx=0x7f21dc01f6c0).stop 2021-03-09T09:34:13.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:13.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:13.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:13.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:13.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:13.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21cc005e30 2021-03-09T09:34:13.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f5d0 con 0x7f21c4005da0 2021-03-09T09:34:13.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21cc0153a0 2021-03-09T09:34:13.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:13.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:13.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:13.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:13.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:13.891 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:13.882+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16832 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:13.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:13.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:13.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:13.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:13.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0145b0 con 0x7f21cc0153a0 2021-03-09T09:34:13.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03c6e0 con 0x7f21cc0153a0 2021-03-09T09:34:13.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 rx=0x7f21dc01f810 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:13.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01f880 con 0x7f21cc0153a0 2021-03-09T09:34:13.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc01f880 con 0x7f21cc0153a0 2021-03-09T09:34:13.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc0153a0 2021-03-09T09:34:13.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.894+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:13.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.898+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:13.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:13.898+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:18.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:18.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:18.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 rx=0x7f21dc01f810 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:18.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 secure :-1 s=READY pgs=5052 cs=0 l=1 rev1=1 rx=0x7f21dc01f810 tx=0x7f21dc07ff90).stop 2021-03-09T09:34:18.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:18.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 unknown :-1 s=CLOSED pgs=5052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:18.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:18.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:18.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:18.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21cc005e30 2021-03-09T09:34:18.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c4005da0 2021-03-09T09:34:18.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01aaa0 con 0x7f21ec0a7f50 2021-03-09T09:34:18.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:18.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:18.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:18.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:18.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:18.898 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16871 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:18.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:18.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:18.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:18.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:18.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0145b0 con 0x7f21ec0a7f50 2021-03-09T09:34:18.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03c6e0 con 0x7f21ec0a7f50 2021-03-09T09:34:18.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc055400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:18.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba030 con 0x7f21ec0a7f50 2021-03-09T09:34:18.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050000 con 0x7f21ec0a7f50 2021-03-09T09:34:18.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21ec0a7f50 2021-03-09T09:34:18.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.898+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:18.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.898+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:18.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:18.898+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:20.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:34:20.878 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:34:23.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:23.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:23.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc055400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:23.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc055400).stop 2021-03-09T09:34:23.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:23.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:23.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:23.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:23.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:23.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21cc005e30 2021-03-09T09:34:23.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041350 con 0x7f21c4005da0 2021-03-09T09:34:23.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0145b0 con 0x7f21cc0153a0 2021-03-09T09:34:23.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:23.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:23.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:23.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:23.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:23.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:23.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:23.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:23.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:23.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089d90 con 0x7f21cc0153a0 2021-03-09T09:34:23.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03c6e0 con 0x7f21cc0153a0 2021-03-09T09:34:23.901 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16906 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:23.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 secure :-1 s=READY pgs=5069 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0191b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:23.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba040 con 0x7f21cc0153a0 2021-03-09T09:34:23.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050000 con 0x7f21cc0153a0 2021-03-09T09:34:23.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.898+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055910 con 0x7f21cc0153a0 2021-03-09T09:34:23.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.898+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:23.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.898+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:23.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:23.898+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:28.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:28.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:28.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 secure :-1 s=READY pgs=5069 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0191b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:28.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 secure :-1 s=READY pgs=5069 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0191b0).stop 2021-03-09T09:34:28.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0898a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:28.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0898a0 unknown :-1 s=CLOSED pgs=5069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:28.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:28.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:28.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:28.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21cc005e30 2021-03-09T09:34:28.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0469e0 con 0x7f21c4005da0 2021-03-09T09:34:28.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0145b0 con 0x7f21ec0a7f50 2021-03-09T09:34:28.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:28.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:28.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:28.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:28.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:28.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:28.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:28.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:28.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:28.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027570 con 0x7f21ec0a7f50 2021-03-09T09:34:28.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089d90 con 0x7f21ec0a7f50 2021-03-09T09:34:28.902 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16945 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:28.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5080 cs=0 l=1 rev1=1 rx=0x7f21dc0b3630 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:28.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba080 con 0x7f21ec0a7f50 2021-03-09T09:34:28.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc068e50 con 0x7f21ec0a7f50 2021-03-09T09:34:28.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21ec0a7f50 2021-03-09T09:34:28.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.901+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:28.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.901+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:28.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:28.901+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:33.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:33.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:33.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5080 cs=0 l=1 rev1=1 rx=0x7f21dc0b3630 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:33.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5080 cs=0 l=1 rev1=1 rx=0x7f21dc0b3630 tx=0x7f21dc080000).stop 2021-03-09T09:34:33.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:33.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:33.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:33.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:33.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:33.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9c0 con 0x7f21cc005e30 2021-03-09T09:34:33.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0411a0 con 0x7f21c4005da0 2021-03-09T09:34:33.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046b80 con 0x7f21cc0153a0 2021-03-09T09:34:33.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:33.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:33.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:33.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:33.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:33.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:33.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:33.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:33.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:33.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027570 con 0x7f21cc0153a0 2021-03-09T09:34:33.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089d90 con 0x7f21cc0153a0 2021-03-09T09:34:33.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc072e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:33.909 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16981 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:33.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:34:33.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3280 con 0x7f21cc0153a0 2021-03-09T09:34:33.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21cc0153a0 2021-03-09T09:34:33.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.905+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:33.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.905+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:33.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:33.905+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:38.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:38.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:38.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc072e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:38.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc072e70).stop 2021-03-09T09:34:38.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:38.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=CLOSED pgs=5090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:38.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:38.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:38.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:38.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015e10 con 0x7f21cc005e30 2021-03-09T09:34:38.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009bb0 con 0x7f21c4005da0 2021-03-09T09:34:38.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053430 con 0x7f21ec0a7f50 2021-03-09T09:34:38.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:38.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:38.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:38.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:38.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:38.909 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17016 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:38.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:38.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:38.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:38.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:38.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027570 con 0x7f21ec0a7f50 2021-03-09T09:34:38.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089d90 con 0x7f21ec0a7f50 2021-03-09T09:34:38.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 rx=0x7f21dc008ff0 tx=0x7f21dc0b6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:38.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b36a0 con 0x7f21ec0a7f50 2021-03-09T09:34:38.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:34:38.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3490 con 0x7f21ec0a7f50 2021-03-09T09:34:38.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.909+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:38.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.909+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:38.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:38.909+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:43.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:43.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:43.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 rx=0x7f21dc008ff0 tx=0x7f21dc0b6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:43.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 rx=0x7f21dc008ff0 tx=0x7f21dc0b6200).stop 2021-03-09T09:34:43.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:43.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:43.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:43.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:43.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:43.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0148e0 con 0x7f21cc005e30 2021-03-09T09:34:43.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042350 con 0x7f21c4005da0 2021-03-09T09:34:43.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046ed0 con 0x7f21cc0153a0 2021-03-09T09:34:43.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:43.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:43.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:43.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:43.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:43.912 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17055 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:43.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:43.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:43.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:43.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:43.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027570 con 0x7f21cc0153a0 2021-03-09T09:34:43.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089d90 con 0x7f21cc0153a0 2021-03-09T09:34:43.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0469e0 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 rx=0x7f21dc05ad70 tx=0x7f21dc065570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:43.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0153a0 2021-03-09T09:34:43.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ab2e0 con 0x7f21cc0153a0 2021-03-09T09:34:43.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc0153a0 2021-03-09T09:34:43.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.917+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:43.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.917+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:43.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:43.917+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:48.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:48.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:48.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0469e0 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 rx=0x7f21dc05ad70 tx=0x7f21dc065570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:48.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0469e0 secure :-1 s=READY pgs=5117 cs=0 l=1 rev1=1 rx=0x7f21dc05ad70 tx=0x7f21dc065570).stop 2021-03-09T09:34:48.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0469e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:48.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0469e0 unknown :-1 s=CLOSED pgs=5117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:48.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:48.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:48.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:48.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027570 con 0x7f21cc005e30 2021-03-09T09:34:48.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d3b0 con 0x7f21c4005da0 2021-03-09T09:34:48.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046ed0 con 0x7f21ec0a7f50 2021-03-09T09:34:48.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:48.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:48.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:48.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:48.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.905+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:48.921 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17089 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:48.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:48.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:48.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:48.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:48.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044db0 con 0x7f21ec0a7f50 2021-03-09T09:34:48.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094090 con 0x7f21ec0a7f50 2021-03-09T09:34:48.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 rx=0x7f21cc056f00 tx=0x7f21e407e350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:48.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:34:48.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:34:48.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:34:48.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.917+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:48.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.921+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:48.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:48.921+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:50.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:34:50.921 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:34:53.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:34:53.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:53.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 rx=0x7f21cc056f00 tx=0x7f21e407e350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5123 cs=0 l=1 rev1=1 rx=0x7f21cc056f00 tx=0x7f21e407e350).stop 2021-03-09T09:34:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:53.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:53.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:53.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0144a0 con 0x7f21cc005e30 2021-03-09T09:34:53.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc051170 con 0x7f21c4005da0 2021-03-09T09:34:53.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015c70 con 0x7f21cc0153a0 2021-03-09T09:34:53.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:53.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:53.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:53.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:53.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:53.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:53.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:53.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:53.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:53.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015db0 con 0x7f21cc0153a0 2021-03-09T09:34:53.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089d90 con 0x7f21cc0153a0 2021-03-09T09:34:53.920 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:53.909+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17129 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:53.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=5135 cs=0 l=1 rev1=1 rx=0x7f21dc0b52e0 tx=0x7f21dc0b5400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:53.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0153a0 2021-03-09T09:34:53.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07b110 con 0x7f21cc0153a0 2021-03-09T09:34:53.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:34:53.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.921+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:53.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.921+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:53.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:53.921+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:34:58.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:58.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:58.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=5135 cs=0 l=1 rev1=1 rx=0x7f21dc0b52e0 tx=0x7f21dc0b5400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:58.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 secure :-1 s=READY pgs=5135 cs=0 l=1 rev1=1 rx=0x7f21dc0b52e0 tx=0x7f21dc0b5400).stop 2021-03-09T09:34:58.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc015780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:58.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc015780 unknown :-1 s=CLOSED pgs=5135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:58.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:58.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:58.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:58.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027570 con 0x7f21cc005e30 2021-03-09T09:34:58.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015d90 con 0x7f21c4005da0 2021-03-09T09:34:58.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046500 con 0x7f21ec0a7f50 2021-03-09T09:34:58.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:34:58.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:58.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:34:58.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:58.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:58.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:58.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:58.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:58.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:58.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046640 con 0x7f21ec0a7f50 2021-03-09T09:34:58.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089d90 con 0x7f21ec0a7f50 2021-03-09T09:34:58.925 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17163 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:34:58.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5144 cs=0 l=1 rev1=1 rx=0x7f21dc021690 tx=0x7f21dc076550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:58.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00d220 con 0x7f21ec0a7f50 2021-03-09T09:34:58.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00dec0 con 0x7f21ec0a7f50 2021-03-09T09:34:58.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc076d80 con 0x7f21ec0a7f50 2021-03-09T09:34:58.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.921+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:34:58.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.925+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:34:58.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:34:58.925+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5144 cs=0 l=1 rev1=1 rx=0x7f21dc021690 tx=0x7f21dc076550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:03.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 secure :-1 s=READY pgs=5144 cs=0 l=1 rev1=1 rx=0x7f21dc021690 tx=0x7f21dc076550).stop 2021-03-09T09:35:03.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc030450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:03.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc030450 unknown :-1 s=CLOSED pgs=5144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:03.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:03.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:03.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:03.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046500 con 0x7f21cc005e30 2021-03-09T09:35:03.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089d90 con 0x7f21c4005da0 2021-03-09T09:35:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0142b0 con 0x7f21cc0153a0 2021-03-09T09:35:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:03.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:03.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:03.926 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17203 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:03.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:03.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:03.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:03.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:03.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0143f0 con 0x7f21cc0153a0 2021-03-09T09:35:03.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046640 con 0x7f21cc0153a0 2021-03-09T09:35:03.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0467d0 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7f21dc028c10 tx=0x7f21dc0764e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:03.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00afd0 con 0x7f21cc0153a0 2021-03-09T09:35:03.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00afd0 con 0x7f21cc0153a0 2021-03-09T09:35:03.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc08a2e0 con 0x7f21cc0153a0 2021-03-09T09:35:03.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.925+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:03.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.925+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:03.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:03.925+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:08.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:08.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:08.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0467d0 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7f21dc028c10 tx=0x7f21dc0764e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:08.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0467d0 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7f21dc028c10 tx=0x7f21dc0764e0).stop 2021-03-09T09:35:08.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc0467d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:08.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc0467d0 unknown :-1 s=CLOSED pgs=5155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:08.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:08.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:08.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:08.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0142b0 con 0x7f21cc005e30 2021-03-09T09:35:08.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0252c0 con 0x7f21c4005da0 2021-03-09T09:35:08.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007ac0 con 0x7f21ec0a7f50 2021-03-09T09:35:08.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:08.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:08.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027570 con 0x7f21ec0a7f50 2021-03-09T09:35:08.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0143f0 con 0x7f21ec0a7f50 2021-03-09T09:35:08.932 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17237 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:08.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.921+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046cc0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 rx=0x7f21dc06a580 tx=0x7f21dc05d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:08.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089360 con 0x7f21ec0a7f50 2021-03-09T09:35:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc098130 con 0x7f21ec0a7f50 2021-03-09T09:35:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.929+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:35:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.929+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:08.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.929+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:08.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:08.929+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:13.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:13.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:13.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046cc0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 rx=0x7f21dc06a580 tx=0x7f21dc05d000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:13.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046cc0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 rx=0x7f21dc06a580 tx=0x7f21dc05d000).stop 2021-03-09T09:35:13.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:13.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046cc0 unknown :-1 s=CLOSED pgs=5165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:13.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:13.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:13.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:13.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e7d0 con 0x7f21cc005e30 2021-03-09T09:35:13.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0140c0 con 0x7f21c4005da0 2021-03-09T09:35:13.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014200 con 0x7f21cc0153a0 2021-03-09T09:35:13.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:13.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:13.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:13.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:13.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:13.933 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17277 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:13.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:13.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:13.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:13.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:13.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025120 con 0x7f21cc0153a0 2021-03-09T09:35:13.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044db0 con 0x7f21cc0153a0 2021-03-09T09:35:13.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 secure :-1 s=READY pgs=5178 cs=0 l=1 rev1=1 rx=0x7f21e40173a0 tx=0x7f21e40b1af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:13.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.929+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40310a0 con 0x7f21cc0153a0 2021-03-09T09:35:13.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.929+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4031200 con 0x7f21cc0153a0 2021-03-09T09:35:13.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.933+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40310a0 con 0x7f21cc0153a0 2021-03-09T09:35:13.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.933+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:13.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.933+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:13.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:13.933+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:18.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc046360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:18.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc046360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:18.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 secure :-1 s=READY pgs=5178 cs=0 l=1 rev1=1 rx=0x7f21e40173a0 tx=0x7f21e40b1af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:18.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 secure :-1 s=READY pgs=5178 cs=0 l=1 rev1=1 rx=0x7f21e40173a0 tx=0x7f21e40b1af0).stop 2021-03-09T09:35:18.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc046360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:18.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 unknown :-1 s=CLOSED pgs=5178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:18.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:18.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:18.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:18.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc005e30 2021-03-09T09:35:18.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027520 con 0x7f21c4005da0 2021-03-09T09:35:18.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21ec0a7f50 2021-03-09T09:35:18.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:18.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:18.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:18.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:18.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:18.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:18.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:18.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:18.937 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17309 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:18.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:18.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046eb0 con 0x7f21ec0a7f50 2021-03-09T09:35:18.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025120 con 0x7f21ec0a7f50 2021-03-09T09:35:18.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e40a4690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:18.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1de0 con 0x7f21ec0a7f50 2021-03-09T09:35:18.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3280 con 0x7f21ec0a7f50 2021-03-09T09:35:18.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3280 con 0x7f21ec0a7f50 2021-03-09T09:35:18.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.936+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:18.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.936+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:18.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:18.936+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:20.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:35:20.970 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:35:23.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:23.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:23.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e40a4690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:23.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e40a4690).stop 2021-03-09T09:35:23.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:23.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=CLOSED pgs=5188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:23.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:23.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:23.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:23.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21cc005e30 2021-03-09T09:35:23.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c410 con 0x7f21c4005da0 2021-03-09T09:35:23.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:35:23.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:23.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:23.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:23.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:23.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:23.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:23.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:23.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:23.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:23.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc014020 con 0x7f21cc0153a0 2021-03-09T09:35:23.941 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17351 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:23.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0143f0 con 0x7f21cc0153a0 2021-03-09T09:35:23.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 secure :-1 s=READY pgs=5197 cs=0 l=1 rev1=1 rx=0x7f21cc0916b0 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:23.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08ceb0 con 0x7f21cc0153a0 2021-03-09T09:35:23.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0153a0 2021-03-09T09:35:23.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:35:23.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.936+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:23.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.940+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:23.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:23.940+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:28.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc046360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:28.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc046360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:28.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 secure :-1 s=READY pgs=5197 cs=0 l=1 rev1=1 rx=0x7f21cc0916b0 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:28.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 secure :-1 s=READY pgs=5197 cs=0 l=1 rev1=1 rx=0x7f21cc0916b0 tx=0x7f21dc009810).stop 2021-03-09T09:35:28.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc046360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:28.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc046360 unknown :-1 s=CLOSED pgs=5197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:28.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:28.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:28.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:28.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21cc005e30 2021-03-09T09:35:28.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049660 con 0x7f21c4005da0 2021-03-09T09:35:28.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:28.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:35:28.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:28.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:28.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:28.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:28.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:28.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:28.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:28.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:28.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027480 con 0x7f21ec0a7f50 2021-03-09T09:35:28.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc014020 con 0x7f21ec0a7f50 2021-03-09T09:35:28.944 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:28.936+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17385 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:28.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5208 cs=0 l=1 rev1=1 rx=0x7f21dc052fd0 tx=0x7f21dc076c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:28.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21ec0a7f50 2021-03-09T09:35:28.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21ec0a7f50 2021-03-09T09:35:28.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:35:28.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.944+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:28.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.944+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:28.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:28.944+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:33.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:33.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:33.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5208 cs=0 l=1 rev1=1 rx=0x7f21dc052fd0 tx=0x7f21dc076c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:33.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5208 cs=0 l=1 rev1=1 rx=0x7f21dc052fd0 tx=0x7f21dc076c20).stop 2021-03-09T09:35:33.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:33.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=CLOSED pgs=5208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:33.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:33.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:33.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:33.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21cc005e30 2021-03-09T09:35:33.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e890 con 0x7f21c4005da0 2021-03-09T09:35:33.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:35:33.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:33.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:33.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:33.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:33.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:33.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:33.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:33.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:33.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:33.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007780 con 0x7f21cc0153a0 2021-03-09T09:35:33.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.936+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046eb0 con 0x7f21cc0153a0 2021-03-09T09:35:33.949 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:33.940+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17425 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:33.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.940+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013e30 secure :-1 s=READY pgs=5218 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40a6bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:33.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21cc0153a0 2021-03-09T09:35:33.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0153a0 2021-03-09T09:35:33.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0153a0 2021-03-09T09:35:33.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.948+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:33.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.948+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:33.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:33.948+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:38.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:38.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:38.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013e30 secure :-1 s=READY pgs=5218 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40a6bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:38.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013e30 secure :-1 s=READY pgs=5218 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40a6bf0).stop 2021-03-09T09:35:38.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:38.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013e30 unknown :-1 s=CLOSED pgs=5218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:38.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:38.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:38.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:38.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21cc005e30 2021-03-09T09:35:38.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21c4005da0 2021-03-09T09:35:38.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0270e0 con 0x7f21ec0a7f50 2021-03-09T09:35:38.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:38.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:38.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:38.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:38.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:38.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:38.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:38.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:38.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:38.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027220 con 0x7f21ec0a7f50 2021-03-09T09:35:38.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc027480 con 0x7f21ec0a7f50 2021-03-09T09:35:38.952 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:38.940+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17459 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:38.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5230 cs=0 l=1 rev1=1 rx=0x7f21dc070b90 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:38.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba070 con 0x7f21ec0a7f50 2021-03-09T09:35:38.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0709d0 con 0x7f21ec0a7f50 2021-03-09T09:35:38.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21ec0a7f50 2021-03-09T09:35:38.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.952+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:38.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.952+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:38.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:38.952+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:43.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:43.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:43.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5230 cs=0 l=1 rev1=1 rx=0x7f21dc070b90 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:43.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5230 cs=0 l=1 rev1=1 rx=0x7f21dc070b90 tx=0x7f21dc010d20).stop 2021-03-09T09:35:43.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:43.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=CLOSED pgs=5230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:43.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:43.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:43.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:43.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21cc005e30 2021-03-09T09:35:43.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0270e0 con 0x7f21c4005da0 2021-03-09T09:35:43.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030140 con 0x7f21cc0153a0 2021-03-09T09:35:43.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:43.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:43.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:43.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:43.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:43.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:43.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:43.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:43.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:43.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:35:43.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc027220 con 0x7f21cc0153a0 2021-03-09T09:35:43.956 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17499 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:43.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013c60 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc06d2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:43.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba070 con 0x7f21cc0153a0 2021-03-09T09:35:43.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0153a0 2021-03-09T09:35:43.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21cc0153a0 2021-03-09T09:35:43.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.952+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:43.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.952+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:43.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:43.952+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:48.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:48.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:48.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013c60 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc06d2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:48.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013c60 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc06d2a0).stop 2021-03-09T09:35:48.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:48.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013c60 unknown :-1 s=CLOSED pgs=5240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:48.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:48.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:48.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:48.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046d40 con 0x7f21cc005e30 2021-03-09T09:35:48.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007b00 con 0x7f21c4005da0 2021-03-09T09:35:48.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027a90 con 0x7f21ec0a7f50 2021-03-09T09:35:48.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:48.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:48.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:48.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:48.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:48.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:48.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:48.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:48.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:48.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027c00 con 0x7f21ec0a7f50 2021-03-09T09:35:48.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007780 con 0x7f21ec0a7f50 2021-03-09T09:35:48.959 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17533 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:48.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5250 cs=0 l=1 rev1=1 rx=0x7f21cc00c6c0 tx=0x7f21e409c0e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:48.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a6e70 con 0x7f21ec0a7f50 2021-03-09T09:35:48.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3070 con 0x7f21ec0a7f50 2021-03-09T09:35:48.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3070 con 0x7f21ec0a7f50 2021-03-09T09:35:48.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.956+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:48.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.960+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:48.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:48.960+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:51.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:35:51.022 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:35:53.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:53.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:53.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5250 cs=0 l=1 rev1=1 rx=0x7f21cc00c6c0 tx=0x7f21e409c0e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:53.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 secure :-1 s=READY pgs=5250 cs=0 l=1 rev1=1 rx=0x7f21cc00c6c0 tx=0x7f21e409c0e0).stop 2021-03-09T09:35:53.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc046850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:53.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc046850 unknown :-1 s=CLOSED pgs=5250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:53.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:53.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:53.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:53.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027d90 con 0x7f21cc005e30 2021-03-09T09:35:53.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014a80 con 0x7f21c4005da0 2021-03-09T09:35:53.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027a90 con 0x7f21cc0153a0 2021-03-09T09:35:53.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:53.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:53.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:53.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:53.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:53.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:53.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:53.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:53.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:53.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007b00 con 0x7f21cc0153a0 2021-03-09T09:35:53.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:35:53.964 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:53.952+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17573 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:53.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013a70 secure :-1 s=READY pgs=5261 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:53.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc0153a0 2021-03-09T09:35:53.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21cc0153a0 2021-03-09T09:35:53.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:35:53.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.964+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:53.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.964+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:53.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:53.964+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:35:58.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:58.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:58.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013a70 secure :-1 s=READY pgs=5261 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:58.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013a70 secure :-1 s=READY pgs=5261 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d90).stop 2021-03-09T09:35:58.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc013a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:58.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc013a70 unknown :-1 s=CLOSED pgs=5261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:58.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:58.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:58.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:58.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027d90 con 0x7f21cc005e30 2021-03-09T09:35:58.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21c4005da0 2021-03-09T09:35:58.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0d0 con 0x7f21ec0a7f50 2021-03-09T09:35:58.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:35:58.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:58.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:35:58.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:58.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:58.966 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:35:58.956+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17607 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:35:58.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:58.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:58.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:58.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:58.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007780 con 0x7f21ec0a7f50 2021-03-09T09:35:58.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007b00 con 0x7f21ec0a7f50 2021-03-09T09:35:58.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026fd0 secure :-1 s=READY pgs=5276 cs=0 l=1 rev1=1 rx=0x7f21dc028670 tx=0x7f21dc01fd10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:58.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0abd70 con 0x7f21ec0a7f50 2021-03-09T09:35:58.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031fc0 con 0x7f21ec0a7f50 2021-03-09T09:35:58.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21ec0a7f50 2021-03-09T09:35:58.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.968+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:35:58.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:35:58.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:35:58.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:03.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:03.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:03.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026fd0 secure :-1 s=READY pgs=5276 cs=0 l=1 rev1=1 rx=0x7f21dc028670 tx=0x7f21dc01fd10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:03.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026fd0 secure :-1 s=READY pgs=5276 cs=0 l=1 rev1=1 rx=0x7f21dc028670 tx=0x7f21dc01fd10).stop 2021-03-09T09:36:03.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:03.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026fd0 unknown :-1 s=CLOSED pgs=5276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:03.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:03.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:03.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:03.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027d90 con 0x7f21cc005e30 2021-03-09T09:36:03.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21c4005da0 2021-03-09T09:36:03.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0d0 con 0x7f21cc0153a0 2021-03-09T09:36:03.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:03.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:03.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:03.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:03.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:03.973 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17647 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:03.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:03.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:03.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:03.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:03.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007b00 con 0x7f21cc0153a0 2021-03-09T09:36:03.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007780 con 0x7f21cc0153a0 2021-03-09T09:36:03.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 secure :-1 s=READY pgs=5286 cs=0 l=1 rev1=1 rx=0x7f21dc00c7b0 tx=0x7f21dc055210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:03.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba070 con 0x7f21cc0153a0 2021-03-09T09:36:03.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005c70 con 0x7f21cc0153a0 2021-03-09T09:36:03.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc0153a0 2021-03-09T09:36:03.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.968+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:03.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.972+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:03.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:03.972+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:08.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:08.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:08.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 secure :-1 s=READY pgs=5286 cs=0 l=1 rev1=1 rx=0x7f21dc00c7b0 tx=0x7f21dc055210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:08.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 secure :-1 s=READY pgs=5286 cs=0 l=1 rev1=1 rx=0x7f21dc00c7b0 tx=0x7f21dc055210).stop 2021-03-09T09:36:08.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:08.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 unknown :-1 s=CLOSED pgs=5286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:08.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:08.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:08.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:08.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027d90 con 0x7f21cc005e30 2021-03-09T09:36:08.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f7f0 con 0x7f21c4005da0 2021-03-09T09:36:08.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21ec0a7f50 2021-03-09T09:36:08.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:08.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:08.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:08.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.960+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:08.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:08.974 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17681 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:08.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:08.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:08.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:08.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:08.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:36:08.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007b00 con 0x7f21ec0a7f50 2021-03-09T09:36:08.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5294 cs=0 l=1 rev1=1 rx=0x7f21dc064340 tx=0x7f21dc070ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:08.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba070 con 0x7f21ec0a7f50 2021-03-09T09:36:08.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0521d0 con 0x7f21ec0a7f50 2021-03-09T09:36:08.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:36:08.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.972+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:08.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.972+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:08.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:08.972+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:13.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:13.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:13.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5294 cs=0 l=1 rev1=1 rx=0x7f21dc064340 tx=0x7f21dc070ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:13.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5294 cs=0 l=1 rev1=1 rx=0x7f21dc064340 tx=0x7f21dc070ab0).stop 2021-03-09T09:36:13.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:13.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=CLOSED pgs=5294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:13.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:13.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:13.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:13.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027d90 con 0x7f21cc005e30 2021-03-09T09:36:13.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21c4005da0 2021-03-09T09:36:13.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0d0 con 0x7f21cc0153a0 2021-03-09T09:36:13.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:13.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:13.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:13.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:13.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:13.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:13.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:13.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:13.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:13.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007780 con 0x7f21cc0153a0 2021-03-09T09:36:13.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:36:13.977 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17721 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:13.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 secure :-1 s=READY pgs=5304 cs=0 l=1 rev1=1 rx=0x7f21dc068370 tx=0x7f21dc028570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:13.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.971+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba070 con 0x7f21cc0153a0 2021-03-09T09:36:13.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.971+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005ea0 con 0x7f21cc0153a0 2021-03-09T09:36:13.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:36:13.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.975+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:13.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:13.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:13.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:18.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:36:18.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:18.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 secure :-1 s=READY pgs=5304 cs=0 l=1 rev1=1 rx=0x7f21dc068370 tx=0x7f21dc028570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:18.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 secure :-1 s=READY pgs=5304 cs=0 l=1 rev1=1 rx=0x7f21dc068370 tx=0x7f21dc028570).stop 2021-03-09T09:36:18.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:18.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050260 unknown :-1 s=CLOSED pgs=5304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:18.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:18.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:18.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:18.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21cc005e30 2021-03-09T09:36:18.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0d0 con 0x7f21c4005da0 2021-03-09T09:36:18.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:18.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:36:18.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:18.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:18.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:18.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:18.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:18.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:18.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:18.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.967+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:18.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.971+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007b00 con 0x7f21ec0a7f50 2021-03-09T09:36:18.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.971+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc027c00 con 0x7f21ec0a7f50 2021-03-09T09:36:18.987 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:18.971+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17755 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:18.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.971+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5315 cs=0 l=1 rev1=1 rx=0x7f21cc030990 tx=0x7f21e40039a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:18.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f9a0 con 0x7f21ec0a7f50 2021-03-09T09:36:18.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f9a0 con 0x7f21ec0a7f50 2021-03-09T09:36:18.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028120 con 0x7f21ec0a7f50 2021-03-09T09:36:18.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.979+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:18.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.983+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:18.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:18.983+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:21.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:36:21.075 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:36:23.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:23.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:23.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5315 cs=0 l=1 rev1=1 rx=0x7f21cc030990 tx=0x7f21e40039a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:23.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5315 cs=0 l=1 rev1=1 rx=0x7f21cc030990 tx=0x7f21e40039a0).stop 2021-03-09T09:36:23.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:23.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=CLOSED pgs=5315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:23.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:23.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:23.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:23.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21cc005e30 2021-03-09T09:36:23.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0d0 con 0x7f21c4005da0 2021-03-09T09:36:23.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:36:23.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:23.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:23.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:23.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:23.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:23.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:23.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:23.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:23.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:23.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f950 con 0x7f21cc0153a0 2021-03-09T09:36:23.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007780 con 0x7f21cc0153a0 2021-03-09T09:36:23.992 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:23.979+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17795 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:23.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 secure :-1 s=READY pgs=5326 cs=0 l=1 rev1=1 rx=0x7f21cc028540 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:23.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099040 con 0x7f21cc0153a0 2021-03-09T09:36:23.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0681d0 con 0x7f21cc0153a0 2021-03-09T09:36:23.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:36:23.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.983+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:23.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.983+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:23.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:23.983+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:28.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc02f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:28.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc02f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:28.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 secure :-1 s=READY pgs=5326 cs=0 l=1 rev1=1 rx=0x7f21cc028540 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:28.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 secure :-1 s=READY pgs=5326 cs=0 l=1 rev1=1 rx=0x7f21cc028540 tx=0x7f21dc010d20).stop 2021-03-09T09:36:28.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc02f460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:28.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 unknown :-1 s=CLOSED pgs=5326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:28.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:28.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:28.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:28.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014c20 con 0x7f21cc005e30 2021-03-09T09:36:28.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ac20 con 0x7f21c4005da0 2021-03-09T09:36:28.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0d0 con 0x7f21ec0a7f50 2021-03-09T09:36:28.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:28.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:28.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:28.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:28.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:28.986 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:28.979+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17829 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:28.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:28.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:28.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:28.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:28.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007780 con 0x7f21ec0a7f50 2021-03-09T09:36:28.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f950 con 0x7f21ec0a7f50 2021-03-09T09:36:28.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5339 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0ad940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:28.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21ec0a7f50 2021-03-09T09:36:28.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6120 con 0x7f21ec0a7f50 2021-03-09T09:36:28.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:36:28.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.991+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:28.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:28.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:28.991+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:33.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:33.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:33.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5339 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0ad940).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:33.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5339 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0ad940).stop 2021-03-09T09:36:33.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:33.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=CLOSED pgs=5339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:33.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:33.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:33.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:33.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028160 con 0x7f21cc005e30 2021-03-09T09:36:33.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc038f60 con 0x7f21c4005da0 2021-03-09T09:36:33.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045230 con 0x7f21cc0153a0 2021-03-09T09:36:33.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:33.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:33.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:33.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:33.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:33.990 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17868 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:33.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:33.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:33.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:33.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:33.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e780 con 0x7f21cc0153a0 2021-03-09T09:36:33.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007b00 con 0x7f21cc0153a0 2021-03-09T09:36:33.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 secure :-1 s=READY pgs=5348 cs=0 l=1 rev1=1 rx=0x7f21e407e340 tx=0x7f21e407e190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:33.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009430 con 0x7f21cc0153a0 2021-03-09T09:36:33.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3280 con 0x7f21cc0153a0 2021-03-09T09:36:33.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3280 con 0x7f21cc0153a0 2021-03-09T09:36:33.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.991+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:33.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.995+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:33.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:33.995+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:38.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc02f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:36:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc02f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 secure :-1 s=READY pgs=5348 cs=0 l=1 rev1=1 rx=0x7f21e407e340 tx=0x7f21e407e190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 secure :-1 s=READY pgs=5348 cs=0 l=1 rev1=1 rx=0x7f21e407e340 tx=0x7f21e407e190).stop 2021-03-09T09:36:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc02f460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc02f460 unknown :-1 s=CLOSED pgs=5348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:38.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:38.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014e30 con 0x7f21cc005e30 2021-03-09T09:36:38.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0b0 con 0x7f21c4005da0 2021-03-09T09:36:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e600 con 0x7f21ec0a7f50 2021-03-09T09:36:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:38.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:38.994 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:38.987+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17903 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:38.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:38.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:38.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:38.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:38.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b2f0 con 0x7f21ec0a7f50 2021-03-09T09:36:38.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e780 con 0x7f21ec0a7f50 2021-03-09T09:36:38.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5362 cs=0 l=1 rev1=1 rx=0x7f21e4017fe0 tx=0x7f21e40a3550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:38.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404a0d0 con 0x7f21ec0a7f50 2021-03-09T09:36:38.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404a0d0 con 0x7f21ec0a7f50 2021-03-09T09:36:38.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400c520 con 0x7f21ec0a7f50 2021-03-09T09:36:38.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.995+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:39.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:39.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:38.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:43.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:43.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:43.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5362 cs=0 l=1 rev1=1 rx=0x7f21e4017fe0 tx=0x7f21e40a3550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:43.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5362 cs=0 l=1 rev1=1 rx=0x7f21e4017fe0 tx=0x7f21e40a3550).stop 2021-03-09T09:36:43.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:43.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=CLOSED pgs=5362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:43.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:43.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:43.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:43.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014e00 con 0x7f21cc005e30 2021-03-09T09:36:43.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0b0 con 0x7f21c4005da0 2021-03-09T09:36:43.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f460 con 0x7f21cc0153a0 2021-03-09T09:36:43.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:43.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:43.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:43.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:43.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:43.996 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17942 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:43.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:43.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:43.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:43.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:43.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f5a0 con 0x7f21cc0153a0 2021-03-09T09:36:43.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b2f0 con 0x7f21cc0153a0 2021-03-09T09:36:43.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b480 secure :-1 s=READY pgs=5369 cs=0 l=1 rev1=1 rx=0x7f21e4053f70 tx=0x7f21e40921e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:44.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40313b0 con 0x7f21cc0153a0 2021-03-09T09:36:44.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc0153a0 2021-03-09T09:36:44.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21cc0153a0 2021-03-09T09:36:44.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.999+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:44.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:44.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:43.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:48.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:48.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:48.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b480 secure :-1 s=READY pgs=5369 cs=0 l=1 rev1=1 rx=0x7f21e4053f70 tx=0x7f21e40921e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:48.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b480 secure :-1 s=READY pgs=5369 cs=0 l=1 rev1=1 rx=0x7f21e4053f70 tx=0x7f21e40921e0).stop 2021-03-09T09:36:48.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00b480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:48.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b480 unknown :-1 s=CLOSED pgs=5369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:48.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:48.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:48.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:48.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0969a0 con 0x7f21cc005e30 2021-03-09T09:36:48.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0b0 con 0x7f21c4005da0 2021-03-09T09:36:48.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014dc0 con 0x7f21ec0a7f50 2021-03-09T09:36:48.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:48.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:48.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:48.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.991+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:48.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:49.000 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17977 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:49.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:49.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:49.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:49.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:49.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00afb0 con 0x7f21ec0a7f50 2021-03-09T09:36:49.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007780 con 0x7f21ec0a7f50 2021-03-09T09:36:49.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.999+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5381 cs=0 l=1 rev1=1 rx=0x7f21dc0b0040 tx=0x7f21dc009340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:49.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068000 con 0x7f21ec0a7f50 2021-03-09T09:36:49.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:48.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc068000 con 0x7f21ec0a7f50 2021-03-09T09:36:49.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:49.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ada00 con 0x7f21ec0a7f50 2021-03-09T09:36:49.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:49.003+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:49.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:49.007+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:49.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:49.007+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:51.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:36:51.128 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:36:53.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:53.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:53.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5381 cs=0 l=1 rev1=1 rx=0x7f21dc0b0040 tx=0x7f21dc009340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:53.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 secure :-1 s=READY pgs=5381 cs=0 l=1 rev1=1 rx=0x7f21dc0b0040 tx=0x7f21dc009340).stop 2021-03-09T09:36:53.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc026f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:54.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc026f80 unknown :-1 s=CLOSED pgs=5381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:54.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:54.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:54.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:54.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0b0 con 0x7f21cc005e30 2021-03-09T09:36:54.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fa40 con 0x7f21c4005da0 2021-03-09T09:36:54.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:54.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fb80 con 0x7f21cc0153a0 2021-03-09T09:36:54.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:54.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:54.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:54.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.995+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:54.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:54.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:54.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:54.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:54.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f730 con 0x7f21cc0153a0 2021-03-09T09:36:54.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f5a0 con 0x7f21cc0153a0 2021-03-09T09:36:54.006 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18016 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:54.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b140 secure :-1 s=READY pgs=5390 cs=0 l=1 rev1=1 rx=0x7f21cc021b30 tx=0x7f21e4091190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:54.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:53.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc0153a0 2021-03-09T09:36:54.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:54.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a43f0 con 0x7f21cc0153a0 2021-03-09T09:36:54.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:54.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0153a0 2021-03-09T09:36:54.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:54.003+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:54.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:54.003+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:54.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:54.003+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:36:59.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:59.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:59.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b140 secure :-1 s=READY pgs=5390 cs=0 l=1 rev1=1 rx=0x7f21cc021b30 tx=0x7f21e4091190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:59.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b140 secure :-1 s=READY pgs=5390 cs=0 l=1 rev1=1 rx=0x7f21cc021b30 tx=0x7f21e4091190).stop 2021-03-09T09:36:59.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00b140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:59.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00b140 unknown :-1 s=CLOSED pgs=5390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:59.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:59.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:59.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:59.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0b0 con 0x7f21cc005e30 2021-03-09T09:36:59.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b5e0 con 0x7f21c4005da0 2021-03-09T09:36:59.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b720 con 0x7f21ec0a7f50 2021-03-09T09:36:59.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:36:59.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:59.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:36:59.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:59.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:58.999+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:59.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:59.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:59.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:59.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:59.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f80 con 0x7f21ec0a7f50 2021-03-09T09:36:59.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00afb0 con 0x7f21ec0a7f50 2021-03-09T09:36:59.009 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18051 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:36:59.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035090 secure :-1 s=READY pgs=5400 cs=0 l=1 rev1=1 rx=0x7f21dc077e80 tx=0x7f21dc0644c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:59.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21ec0a7f50 2021-03-09T09:36:59.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21ec0a7f50 2021-03-09T09:36:59.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.007+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:36:59.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.007+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:36:59.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.011+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:36:59.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:36:59.011+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:04.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc035090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:04.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc035090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:04.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035090 secure :-1 s=READY pgs=5400 cs=0 l=1 rev1=1 rx=0x7f21dc077e80 tx=0x7f21dc0644c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:04.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035090 secure :-1 s=READY pgs=5400 cs=0 l=1 rev1=1 rx=0x7f21dc077e80 tx=0x7f21dc0644c0).stop 2021-03-09T09:37:04.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc035090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:04.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc035090 unknown :-1 s=CLOSED pgs=5400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:04.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:04.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:04.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:04.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a660 con 0x7f21cc005e30 2021-03-09T09:37:04.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b860 con 0x7f21c4005da0 2021-03-09T09:37:04.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027110 con 0x7f21cc0153a0 2021-03-09T09:37:04.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:04.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:04.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:04.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:04.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.003+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:04.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:04.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:04.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:04.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:04.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027280 con 0x7f21cc0153a0 2021-03-09T09:37:04.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f730 con 0x7f21cc0153a0 2021-03-09T09:37:04.014 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18090 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:04.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.007+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00afb0 secure :-1 s=READY pgs=5413 cs=0 l=1 rev1=1 rx=0x7f21cc057040 tx=0x7f21e4048730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:04.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40553b0 con 0x7f21cc0153a0 2021-03-09T09:37:04.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40553b0 con 0x7f21cc0153a0 2021-03-09T09:37:04.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.015+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21cc0153a0 2021-03-09T09:37:04.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.015+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:04.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.015+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:04.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:04.015+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:09.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:09.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:09.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00afb0 secure :-1 s=READY pgs=5413 cs=0 l=1 rev1=1 rx=0x7f21cc057040 tx=0x7f21e4048730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:09.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00afb0 secure :-1 s=READY pgs=5413 cs=0 l=1 rev1=1 rx=0x7f21cc057040 tx=0x7f21e4048730).stop 2021-03-09T09:37:09.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00afb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:09.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00afb0 unknown :-1 s=CLOSED pgs=5413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:09.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:09.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:09.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:09.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fd80 con 0x7f21cc005e30 2021-03-09T09:37:09.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a660 con 0x7f21c4005da0 2021-03-09T09:37:09.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027110 con 0x7f21ec0a7f50 2021-03-09T09:37:09.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:09.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:09.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:09.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:09.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.006+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:09.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:09.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:09.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:09.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:09.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b7e0 con 0x7f21ec0a7f50 2021-03-09T09:37:09.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026f80 con 0x7f21ec0a7f50 2021-03-09T09:37:09.016 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18125 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:09.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034ea0 secure :-1 s=READY pgs=5424 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:09.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21ec0a7f50 2021-03-09T09:37:09.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:37:09.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:37:09.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.018+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:09.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.018+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:09.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:09.018+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:14.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc034ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:14.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc034ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:14.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034ea0 secure :-1 s=READY pgs=5424 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:14.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034ea0 secure :-1 s=READY pgs=5424 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc02df50).stop 2021-03-09T09:37:14.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc034ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:14.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034ea0 unknown :-1 s=CLOSED pgs=5424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:14.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:14.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:14.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:14.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027120 con 0x7f21cc005e30 2021-03-09T09:37:14.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b860 con 0x7f21c4005da0 2021-03-09T09:37:14.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a630 con 0x7f21cc0153a0 2021-03-09T09:37:14.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:14.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:14.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:14.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:14.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:14.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:14.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:14.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:14.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:14.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02fd30 con 0x7f21cc0153a0 2021-03-09T09:37:14.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.010+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b7e0 con 0x7f21cc0153a0 2021-03-09T09:37:14.021 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:14.014+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18164 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:14.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5432 cs=0 l=1 rev1=1 rx=0x7f21dc081a40 tx=0x7f21dc0ba710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:14.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02fc40 con 0x7f21cc0153a0 2021-03-09T09:37:14.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02fc40 con 0x7f21cc0153a0 2021-03-09T09:37:14.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bad80 con 0x7f21cc0153a0 2021-03-09T09:37:14.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.022+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:14.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.022+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:14.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:14.022+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:19.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:19.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:19.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5432 cs=0 l=1 rev1=1 rx=0x7f21dc081a40 tx=0x7f21dc0ba710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:19.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5432 cs=0 l=1 rev1=1 rx=0x7f21dc081a40 tx=0x7f21dc0ba710).stop 2021-03-09T09:37:19.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:19.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:19.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:19.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:19.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:19.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f80 con 0x7f21cc005e30 2021-03-09T09:37:19.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0270c0 con 0x7f21c4005da0 2021-03-09T09:37:19.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0351a0 con 0x7f21ec0a7f50 2021-03-09T09:37:19.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:19.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:19.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:19.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:19.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:19.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:19.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:19.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:19.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:19.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc035310 con 0x7f21ec0a7f50 2021-03-09T09:37:19.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.014+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc027280 con 0x7f21ec0a7f50 2021-03-09T09:37:19.026 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:19.018+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18199 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:19.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034cb0 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 rx=0x7f21cc0148b0 tx=0x7f21e4020fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:19.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:37:19.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009390 con 0x7f21ec0a7f50 2021-03-09T09:37:19.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:37:19.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.022+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:19.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:19.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:19.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:21.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:37:21.190 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:37:24.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc034cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:24.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc034cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:24.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034cb0 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 rx=0x7f21cc0148b0 tx=0x7f21e4020fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:24.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034cb0 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 rx=0x7f21cc0148b0 tx=0x7f21e4020fe0).stop 2021-03-09T09:37:24.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc034cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:24.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc034cb0 unknown :-1 s=CLOSED pgs=5443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:24.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:24.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:24.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:24.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc005e30 2021-03-09T09:37:24.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089f00 con 0x7f21c4005da0 2021-03-09T09:37:24.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0351a0 con 0x7f21cc0153a0 2021-03-09T09:37:24.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:24.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:24.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:24.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:24.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:24.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:24.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:24.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:24.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:24.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b300 con 0x7f21cc0153a0 2021-03-09T09:37:24.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc035310 con 0x7f21cc0153a0 2021-03-09T09:37:24.028 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:24.018+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18238 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:24.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.022+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5454 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4036450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:24.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21cc0153a0 2021-03-09T09:37:24.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4048180 con 0x7f21cc0153a0 2021-03-09T09:37:24.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40482f0 con 0x7f21cc0153a0 2021-03-09T09:37:24.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.026+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:24.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:24.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:24.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:29.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:29.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:29.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5454 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4036450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:29.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5454 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4036450).stop 2021-03-09T09:37:29.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:29.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:29.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:29.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:29.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:29.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc005e30 2021-03-09T09:37:29.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a0a0 con 0x7f21c4005da0 2021-03-09T09:37:29.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092160 con 0x7f21ec0a7f50 2021-03-09T09:37:29.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:29.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:29.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:29.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:29.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:29.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:29.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:29.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:29.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:29.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b300 con 0x7f21ec0a7f50 2021-03-09T09:37:29.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02fd30 con 0x7f21ec0a7f50 2021-03-09T09:37:29.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:29.033 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18273 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:29.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0abeb0 con 0x7f21ec0a7f50 2021-03-09T09:37:29.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0624b0 con 0x7f21ec0a7f50 2021-03-09T09:37:29.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21ec0a7f50 2021-03-09T09:37:29.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.030+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:29.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.030+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:29.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:29.030+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:34.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:34.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:34.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:34.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc08a120).stop 2021-03-09T09:37:34.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:34.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 unknown :-1 s=CLOSED pgs=5464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:34.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:34.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:34.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:34.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc005e30 2021-03-09T09:37:34.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0412a0 con 0x7f21c4005da0 2021-03-09T09:37:34.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae10 con 0x7f21cc0153a0 2021-03-09T09:37:34.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:34.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:34.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:34.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:34.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:34.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:34.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:34.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:34.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:34.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00af50 con 0x7f21cc0153a0 2021-03-09T09:37:34.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc035310 con 0x7f21cc0153a0 2021-03-09T09:37:34.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089d60 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 rx=0x7f21cc02cce0 tx=0x7f21e4005480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:34.035 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18312 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:34.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21cc0153a0 2021-03-09T09:37:34.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407dc80 con 0x7f21cc0153a0 2021-03-09T09:37:34.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407dc80 con 0x7f21cc0153a0 2021-03-09T09:37:34.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.034+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:34.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.034+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:34.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:34.034+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:39.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc089d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:39.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc089d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:39.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089d60 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 rx=0x7f21cc02cce0 tx=0x7f21e4005480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:39.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089d60 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 rx=0x7f21cc02cce0 tx=0x7f21e4005480).stop 2021-03-09T09:37:39.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc089d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:39.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc089d60 unknown :-1 s=CLOSED pgs=5475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:39.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:39.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:39.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:39.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:39.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:39.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc005e30 2021-03-09T09:37:39.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae10 con 0x7f21c4005da0 2021-03-09T09:37:39.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0412a0 con 0x7f21ec0a7f50 2021-03-09T09:37:39.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.030+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:39.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.030+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:39.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.030+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:39.038 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:39.030+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18347 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:39.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:39.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:39.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:39.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:39.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f10 con 0x7f21ec0a7f50 2021-03-09T09:37:39.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00af50 con 0x7f21ec0a7f50 2021-03-09T09:37:39.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 secure :-1 s=READY pgs=5491 cs=0 l=1 rev1=1 rx=0x7f21e408a700 tx=0x7f21e40a4510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:39.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4086630 con 0x7f21ec0a7f50 2021-03-09T09:37:39.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4086630 con 0x7f21ec0a7f50 2021-03-09T09:37:39.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.038+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21ec0a7f50 2021-03-09T09:37:39.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.038+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:39.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.042+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:39.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:39.042+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:44.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:44.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:44.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 secure :-1 s=READY pgs=5491 cs=0 l=1 rev1=1 rx=0x7f21e408a700 tx=0x7f21e40a4510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:44.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 secure :-1 s=READY pgs=5491 cs=0 l=1 rev1=1 rx=0x7f21e408a700 tx=0x7f21e40a4510).stop 2021-03-09T09:37:44.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:44.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050200 unknown :-1 s=CLOSED pgs=5491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:44.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:44.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:44.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:44.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc005e30 2021-03-09T09:37:44.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046c90 con 0x7f21c4005da0 2021-03-09T09:37:44.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050f80 con 0x7f21cc0153a0 2021-03-09T09:37:44.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:44.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:44.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.030+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:44.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:44.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:44.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:44.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:44.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:44.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:44.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0412a0 con 0x7f21cc0153a0 2021-03-09T09:37:44.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02fd30 con 0x7f21cc0153a0 2021-03-09T09:37:44.043 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:44.034+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18386 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:44.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.038+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5497 cs=0 l=1 rev1=1 rx=0x7f21dc06db90 tx=0x7f21dc068000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:44.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.038+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc0153a0 2021-03-09T09:37:44.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.038+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc0153a0 2021-03-09T09:37:44.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc0153a0 2021-03-09T09:37:44.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.046+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:44.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.046+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:44.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:44.046+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:49.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:49.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:49.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5497 cs=0 l=1 rev1=1 rx=0x7f21dc06db90 tx=0x7f21dc068000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:49.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5497 cs=0 l=1 rev1=1 rx=0x7f21dc06db90 tx=0x7f21dc068000).stop 2021-03-09T09:37:49.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:49.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:49.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:49.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:49.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:49.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc005e30 2021-03-09T09:37:49.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046c90 con 0x7f21c4005da0 2021-03-09T09:37:49.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0951c0 con 0x7f21ec0a7f50 2021-03-09T09:37:49.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.034+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:49.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:49.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:49.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:49.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:49.046 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18421 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:49.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:49.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:49.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:49.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:49.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0412a0 con 0x7f21ec0a7f50 2021-03-09T09:37:49.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026f10 con 0x7f21ec0a7f50 2021-03-09T09:37:49.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.038+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089ce0 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 rx=0x7f21e40b0ff0 tx=0x7f21e400ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:49.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.042+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4048590 con 0x7f21ec0a7f50 2021-03-09T09:37:49.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.042+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408fca0 con 0x7f21ec0a7f50 2021-03-09T09:37:49.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21ec0a7f50 2021-03-09T09:37:49.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.046+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:49.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.046+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:49.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:49.046+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:51.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:37:51.238 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:37:54.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:54.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:54.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089ce0 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 rx=0x7f21e40b0ff0 tx=0x7f21e400ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:54.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089ce0 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 rx=0x7f21e40b0ff0 tx=0x7f21e400ff10).stop 2021-03-09T09:37:54.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc089ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:54.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc089ce0 unknown :-1 s=CLOSED pgs=5508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:54.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:54.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:54.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:54.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015b90 con 0x7f21cc005e30 2021-03-09T09:37:54.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046c90 con 0x7f21c4005da0 2021-03-09T09:37:54.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fe30 con 0x7f21cc0153a0 2021-03-09T09:37:54.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:54.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:54.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:54.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:54.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:54.048 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18460 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:54.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:54.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:54.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:54.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:54.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0412a0 con 0x7f21cc0153a0 2021-03-09T09:37:54.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02fd30 con 0x7f21cc0153a0 2021-03-09T09:37:54.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.042+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc052a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:54.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.050+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b31f0 con 0x7f21cc0153a0 2021-03-09T09:37:54.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b31f0 con 0x7f21cc0153a0 2021-03-09T09:37:54.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0153a0 2021-03-09T09:37:54.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.054+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:54.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.054+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:54.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:54.054+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:37:59.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:59.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:59.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc052a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:59.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc052a70).stop 2021-03-09T09:37:59.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:59.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:59.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:59.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:59.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:59.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0412a0 con 0x7f21cc005e30 2021-03-09T09:37:59.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089ce0 con 0x7f21c4005da0 2021-03-09T09:37:59.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089e20 con 0x7f21ec0a7f50 2021-03-09T09:37:59.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:37:59.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:59.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:37:59.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:59.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:59.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:59.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:59.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:59.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:59.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089f90 con 0x7f21ec0a7f50 2021-03-09T09:37:59.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02fd30 con 0x7f21ec0a7f50 2021-03-09T09:37:59.054 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18495 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:37:59.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095160 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 rx=0x7f21cc014b60 tx=0x7f21dc070dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:59.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21ec0a7f50 2021-03-09T09:37:59.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.046+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017160 con 0x7f21ec0a7f50 2021-03-09T09:37:59.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.054+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21ec0a7f50 2021-03-09T09:37:59.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.054+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:37:59.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.054+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:37:59.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:37:59.054+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:04.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc095160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:04.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc095160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:04.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095160 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 rx=0x7f21cc014b60 tx=0x7f21dc070dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:04.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095160 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 rx=0x7f21cc014b60 tx=0x7f21dc070dd0).stop 2021-03-09T09:38:04.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc095160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:04.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc095160 unknown :-1 s=CLOSED pgs=5528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:04.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:04.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:04.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:04.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b300 con 0x7f21cc005e30 2021-03-09T09:38:04.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fce0 con 0x7f21c4005da0 2021-03-09T09:38:04.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fe20 con 0x7f21cc0153a0 2021-03-09T09:38:04.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:04.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:04.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:04.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:04.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.045+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:04.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:04.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:04.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:04.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:04.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0242a0 con 0x7f21cc0153a0 2021-03-09T09:38:04.057 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18534 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:04.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089f90 con 0x7f21cc0153a0 2021-03-09T09:38:04.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5540 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc069690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:04.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc0153a0 2021-03-09T09:38:04.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017210 con 0x7f21cc0153a0 2021-03-09T09:38:04.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc064000 con 0x7f21cc0153a0 2021-03-09T09:38:04.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.057+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:04.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.057+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:04.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:04.057+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:09.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:09.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:09.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5540 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc069690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:09.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5540 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc069690).stop 2021-03-09T09:38:09.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:09.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:09.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:09.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:09.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:09.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091570 con 0x7f21cc005e30 2021-03-09T09:38:09.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c9c0 con 0x7f21c4005da0 2021-03-09T09:38:09.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f70 con 0x7f21ec0a7f50 2021-03-09T09:38:09.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:09.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:09.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:09.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:09.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:09.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:09.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:09.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:09.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:09.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0950b0 con 0x7f21ec0a7f50 2021-03-09T09:38:09.061 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18569 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:09.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0242a0 con 0x7f21ec0a7f50 2021-03-09T09:38:09.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fcc0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 rx=0x7f21dc0204c0 tx=0x7f21dc093f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:09.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21ec0a7f50 2021-03-09T09:38:09.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06ac30 con 0x7f21ec0a7f50 2021-03-09T09:38:09.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21ec0a7f50 2021-03-09T09:38:09.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.061+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:09.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.061+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:09.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:09.061+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:14.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:14.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:14.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fcc0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 rx=0x7f21dc0204c0 tx=0x7f21dc093f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:14.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fcc0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 rx=0x7f21dc0204c0 tx=0x7f21dc093f30).stop 2021-03-09T09:38:14.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fcc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:14.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fcc0 unknown :-1 s=CLOSED pgs=5552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:14.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045050 con 0x7f21cc005e30 2021-03-09T09:38:14.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b300 con 0x7f21c4005da0 2021-03-09T09:38:14.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024230 con 0x7f21cc0153a0 2021-03-09T09:38:14.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:14.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:14.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:14.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:14.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:14.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.053+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:14.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:14.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:14.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:14.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0243a0 con 0x7f21cc0153a0 2021-03-09T09:38:14.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0950b0 con 0x7f21cc0153a0 2021-03-09T09:38:14.069 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18608 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:14.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f21dc0ac020 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:14.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc0153a0 2021-03-09T09:38:14.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc0153a0 2021-03-09T09:38:14.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc064000 con 0x7f21cc0153a0 2021-03-09T09:38:14.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.061+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:14.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.065+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:14.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:14.065+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f21dc0ac020 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f21dc0ac020 tx=0x7f21dc009810).stop 2021-03-09T09:38:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045050 con 0x7f21cc005e30 2021-03-09T09:38:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0241e0 con 0x7f21c4005da0 2021-03-09T09:38:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0950b0 con 0x7f21ec0a7f50 2021-03-09T09:38:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:19.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:19.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:19.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:19.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:19.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:19.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:19.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:19.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:19.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095220 con 0x7f21ec0a7f50 2021-03-09T09:38:19.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026f10 con 0x7f21ec0a7f50 2021-03-09T09:38:19.068 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18643 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:19.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5571 cs=0 l=1 rev1=1 rx=0x7f21e4046cc0 tx=0x7f21e402b270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:19.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402b3d0 con 0x7f21ec0a7f50 2021-03-09T09:38:19.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402b550 con 0x7f21ec0a7f50 2021-03-09T09:38:19.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40362d0 con 0x7f21ec0a7f50 2021-03-09T09:38:19.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.069+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:19.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.069+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:19.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:19.069+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:21.281 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:21.292 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:24.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:24.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:24.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5571 cs=0 l=1 rev1=1 rx=0x7f21e4046cc0 tx=0x7f21e402b270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:24.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5571 cs=0 l=1 rev1=1 rx=0x7f21e4046cc0 tx=0x7f21e402b270).stop 2021-03-09T09:38:24.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:24.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=CLOSED pgs=5571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:24.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:24.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:24.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:24.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b300 con 0x7f21cc005e30 2021-03-09T09:38:24.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21c4005da0 2021-03-09T09:38:24.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0954f0 con 0x7f21cc0153a0 2021-03-09T09:38:24.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:24.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:24.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:24.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc050120 con 0x7f21cc0153a0 2021-03-09T09:38:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.061+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0243a0 con 0x7f21cc0153a0 2021-03-09T09:38:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5582 cs=0 l=1 rev1=1 rx=0x7f21dc00d560 tx=0x7f21dc070ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:24.072 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:24.065+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18682 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:24.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc070f30 con 0x7f21cc0153a0 2021-03-09T09:38:24.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0153a0 2021-03-09T09:38:24.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc064000 con 0x7f21cc0153a0 2021-03-09T09:38:24.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.073+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:24.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.073+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:24.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:24.073+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:29.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:29.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:29.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5582 cs=0 l=1 rev1=1 rx=0x7f21dc00d560 tx=0x7f21dc070ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:29.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5582 cs=0 l=1 rev1=1 rx=0x7f21dc00d560 tx=0x7f21dc070ec0).stop 2021-03-09T09:38:29.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:29.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:29.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:29.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:29.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:29.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045270 con 0x7f21cc005e30 2021-03-09T09:38:29.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f10 con 0x7f21c4005da0 2021-03-09T09:38:29.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21ec0a7f50 2021-03-09T09:38:29.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:29.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:29.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:29.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:29.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:29.074 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18716 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:29.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:29.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:29.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:29.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:29.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095520 con 0x7f21ec0a7f50 2021-03-09T09:38:29.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc050120 con 0x7f21ec0a7f50 2021-03-09T09:38:29.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5596 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0ab1b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:29.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21ec0a7f50 2021-03-09T09:38:29.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017eb0 con 0x7f21ec0a7f50 2021-03-09T09:38:29.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062dc0 con 0x7f21ec0a7f50 2021-03-09T09:38:29.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.077+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:29.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.077+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:29.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:29.077+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:34.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:34.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:34.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5596 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0ab1b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:34.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5596 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0ab1b0).stop 2021-03-09T09:38:34.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:34.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=CLOSED pgs=5596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:34.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:34.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:34.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:34.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a100 con 0x7f21cc005e30 2021-03-09T09:38:34.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21c4005da0 2021-03-09T09:38:34.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045270 con 0x7f21cc0153a0 2021-03-09T09:38:34.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:34.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:34.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:34.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:34.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:34.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:34.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:34.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:34.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:34.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b300 con 0x7f21cc0153a0 2021-03-09T09:38:34.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095520 con 0x7f21cc0153a0 2021-03-09T09:38:34.088 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:34.069+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:34.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5606 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:34.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0153a0 2021-03-09T09:38:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc0153a0 2021-03-09T09:38:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc0153a0 2021-03-09T09:38:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.077+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:34.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:34.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:39.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:39.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:39.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5606 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:39.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 secure :-1 s=READY pgs=5606 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc009810).stop 2021-03-09T09:38:39.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc00ae10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:39.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc00ae10 unknown :-1 s=CLOSED pgs=5606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:39.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:39.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:39.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:39.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:38:39.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21c4005da0 2021-03-09T09:38:39.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b300 con 0x7f21ec0a7f50 2021-03-09T09:38:39.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:39.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:39.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:39.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:39.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:39.082 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18790 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:39.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:39.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:39.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:39.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:39.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f10 con 0x7f21ec0a7f50 2021-03-09T09:38:39.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095220 con 0x7f21ec0a7f50 2021-03-09T09:38:39.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5615 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:39.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f1b0 con 0x7f21ec0a7f50 2021-03-09T09:38:39.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f1b0 con 0x7f21ec0a7f50 2021-03-09T09:38:39.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T09:38:39.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.085+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:39.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.085+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:39.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:39.085+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:44.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:44.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:44.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5615 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:44.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 secure :-1 s=READY pgs=5615 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4005180).stop 2021-03-09T09:38:44.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:44.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc90 unknown :-1 s=CLOSED pgs=5615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:44.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:44.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:44.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:44.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21cc005e30 2021-03-09T09:38:44.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b90 con 0x7f21c4005da0 2021-03-09T09:38:44.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc0153a0 2021-03-09T09:38:44.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:44.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:44.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:44.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:44.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:44.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:44.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:44.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:44.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:44.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f10 con 0x7f21cc0153a0 2021-03-09T09:38:44.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095220 con 0x7f21cc0153a0 2021-03-09T09:38:44.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.077+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050120 secure :-1 s=READY pgs=5624 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e40b1960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:44.088 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:44.081+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18830 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:44.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400c4c0 con 0x7f21cc0153a0 2021-03-09T09:38:44.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400c620 con 0x7f21cc0153a0 2021-03-09T09:38:44.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400c4c0 con 0x7f21cc0153a0 2021-03-09T09:38:44.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.085+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:44.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.085+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:44.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:44.085+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:49.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:49.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:49.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050120 secure :-1 s=READY pgs=5624 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e40b1960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:49.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050120 secure :-1 s=READY pgs=5624 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e40b1960).stop 2021-03-09T09:38:49.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:49.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050120 unknown :-1 s=CLOSED pgs=5624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:49.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:49.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21cc005e30 2021-03-09T09:38:49.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4005da0 2021-03-09T09:38:49.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026e80 con 0x7f21ec0a7f50 2021-03-09T09:38:49.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:49.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:49.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:49.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:49.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:49.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:49.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:49.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:49.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:49.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc050610 con 0x7f21ec0a7f50 2021-03-09T09:38:49.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095520 con 0x7f21ec0a7f50 2021-03-09T09:38:49.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc70 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 rx=0x7f21dc0726b0 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:49.093 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:49.085+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18864 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:49.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0726e0 con 0x7f21ec0a7f50 2021-03-09T09:38:49.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21ec0a7f50 2021-03-09T09:38:49.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:38:49.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.089+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:49.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:49.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:49.089+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:51.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:51.358 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:38:54.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:54.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:54.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc70 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 rx=0x7f21dc0726b0 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:54.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc70 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 rx=0x7f21dc0726b0 tx=0x7f21dc02dfc0).stop 2021-03-09T09:38:54.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:54.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc70 unknown :-1 s=CLOSED pgs=5635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:54.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:54.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:54.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:54.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953b0 con 0x7f21cc005e30 2021-03-09T09:38:54.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030250 con 0x7f21c4005da0 2021-03-09T09:38:54.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc0153a0 2021-03-09T09:38:54.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:54.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:54.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:54.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:54.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:54.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:54.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:54.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:54.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:54.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0956c0 con 0x7f21cc0153a0 2021-03-09T09:38:54.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.085+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc050610 con 0x7f21cc0153a0 2021-03-09T09:38:54.097 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:54.089+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18904 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:54.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.089+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050080 secure :-1 s=READY pgs=5647 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc06ad90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:54.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02b230 con 0x7f21cc0153a0 2021-03-09T09:38:54.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02b390 con 0x7f21cc0153a0 2021-03-09T09:38:54.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21cc0153a0 2021-03-09T09:38:54.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.093+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:54.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:54.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:54.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:38:59.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:59.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:59.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050080 secure :-1 s=READY pgs=5647 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc06ad90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:59.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050080 secure :-1 s=READY pgs=5647 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc06ad90).stop 2021-03-09T09:38:59.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc050080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:59.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc050080 unknown :-1 s=CLOSED pgs=5647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:59.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:59.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:59.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:59.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030250 con 0x7f21cc005e30 2021-03-09T09:38:59.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21c4005da0 2021-03-09T09:38:59.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f20 con 0x7f21ec0a7f50 2021-03-09T09:38:59.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:38:59.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:59.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:38:59.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:59.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:59.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:59.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:59.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:59.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:59.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e440 con 0x7f21ec0a7f50 2021-03-09T09:38:59.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095220 con 0x7f21ec0a7f50 2021-03-09T09:38:59.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc50 secure :-1 s=READY pgs=5656 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e407e450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:59.100 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:38:59.088+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18938 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:38:59.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21ec0a7f50 2021-03-09T09:38:59.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21ec0a7f50 2021-03-09T09:38:59.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21ec0a7f50 2021-03-09T09:38:59.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.096+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:38:59.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.096+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:38:59.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:38:59.096+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:04.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:04.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:04.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc50 secure :-1 s=READY pgs=5656 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e407e450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:04.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc50 secure :-1 s=READY pgs=5656 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e407e450).stop 2021-03-09T09:39:04.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc02fc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:04.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc02fc50 unknown :-1 s=CLOSED pgs=5656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:04.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:04.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:04.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:04.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089f20 con 0x7f21cc005e30 2021-03-09T09:39:04.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a060 con 0x7f21c4005da0 2021-03-09T09:39:04.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050080 con 0x7f21cc0153a0 2021-03-09T09:39:04.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:04.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:04.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:04.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:04.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:04.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 msgr2=0x7f21cc095120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:39:04.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0153a0 0x7f21cc095120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:04.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01adc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:04.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4005da0 0x7f21cc01adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:04.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc03ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:04.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc03ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:04.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21c40037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:04.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21c40081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:04.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c4004110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:04.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089f20 con 0x7f21cc005e30 2021-03-09T09:39:04.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a060 con 0x7f21c4003cb0 2021-03-09T09:39:04.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050080 con 0x7f21c4008860 2021-03-09T09:39:04.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21c40081b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:04.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21c40081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:04.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21c4004110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:04.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c4004110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:04.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21c40037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:04.112 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:04.092+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18978 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:04.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21c40081b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:04.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21c40081b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:04.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21c4004110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:04.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c4004110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:04.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0501c0 con 0x7f21c4008860 2021-03-09T09:39:04.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0956c0 con 0x7f21c4008860 2021-03-09T09:39:04.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21c40037c0 secure :-1 s=READY pgs=5669 cs=0 l=1 rev1=1 rx=0x7f21cc02d940 tx=0x7f21dc0ab0d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:04.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc028e70 con 0x7f21c4008860 2021-03-09T09:39:04.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4008860 2021-03-09T09:39:04.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21c4008860 2021-03-09T09:39:04.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.104+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:04.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:04.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:04.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:09.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21c40037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:09.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.092+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21c40037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:09.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21c40037c0 secure :-1 s=READY pgs=5669 cs=0 l=1 rev1=1 rx=0x7f21cc02d940 tx=0x7f21dc0ab0d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:09.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.092+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21c40037c0 secure :-1 s=READY pgs=5669 cs=0 l=1 rev1=1 rx=0x7f21cc02d940 tx=0x7f21dc0ab0d0).stop 2021-03-09T09:39:09.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21c40037c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:09.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21c40037c0 unknown :-1 s=CLOSED pgs=5669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:09.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:09.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:09.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:09.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050080 con 0x7f21cc005e30 2021-03-09T09:39:09.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004970 con 0x7f21c4003cb0 2021-03-09T09:39:09.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21ec0a7f50 2021-03-09T09:39:09.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc095120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:09.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc095120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:09.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:09.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:09.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:09.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:09.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:09.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc095120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:09.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc095120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:09.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f370 con 0x7f21ec0a7f50 2021-03-09T09:39:09.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e440 con 0x7f21ec0a7f50 2021-03-09T09:39:09.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ee80 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 rx=0x7f21cc045140 tx=0x7f21e4016b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:09.112 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:09.096+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19012 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:09.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21ec0a7f50 2021-03-09T09:39:09.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009390 con 0x7f21ec0a7f50 2021-03-09T09:39:09.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:39:09.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.104+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:09.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:09.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:09.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:14.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:14.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:14.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ee80 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 rx=0x7f21cc045140 tx=0x7f21e4016b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:14.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ee80 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 rx=0x7f21cc045140 tx=0x7f21e4016b20).stop 2021-03-09T09:39:14.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ee80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:14.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ee80 unknown :-1 s=CLOSED pgs=5678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:14.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:14.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:14.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:14.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050080 con 0x7f21cc005e30 2021-03-09T09:39:14.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027170 con 0x7f21c4003cb0 2021-03-09T09:39:14.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e5e0 con 0x7f21c4008860 2021-03-09T09:39:14.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:14.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:14.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:14.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:14.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:14.109 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19052 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:14.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:14.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:14.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:14.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:14.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b0e0 con 0x7f21c4008860 2021-03-09T09:39:14.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.100+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0501c0 con 0x7f21c4008860 2021-03-09T09:39:14.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc095120 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:14.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4008860 2021-03-09T09:39:14.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4008860 2021-03-09T09:39:14.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc055000 con 0x7f21c4008860 2021-03-09T09:39:14.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.112+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:14.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.112+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:14.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:14.112+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:19.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc095120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:19.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc095120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:19.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc095120 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:19.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc095120 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d60).stop 2021-03-09T09:39:19.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc095120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:19.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc095120 unknown :-1 s=CLOSED pgs=5692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:19.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:19.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:19.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:19.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0300a0 con 0x7f21cc005e30 2021-03-09T09:39:19.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027170 con 0x7f21c4003cb0 2021-03-09T09:39:19.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e5e0 con 0x7f21ec0a7f50 2021-03-09T09:39:19.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:19.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:19.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:19.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:19.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:19.113 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19086 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:19.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:19.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:19.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:19.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:19.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026de0 con 0x7f21ec0a7f50 2021-03-09T09:39:19.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.104+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f370 con 0x7f21ec0a7f50 2021-03-09T09:39:19.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ede0 secure :-1 s=READY pgs=5700 cs=0 l=1 rev1=1 rx=0x7f21cc0433a0 tx=0x7f21e40a6df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:19.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407e2e0 con 0x7f21ec0a7f50 2021-03-09T09:39:19.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21ec0a7f50 2021-03-09T09:39:19.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21ec0a7f50 2021-03-09T09:39:19.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.112+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:19.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:19.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:19.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:21.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:21.414 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:24.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:24.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:24.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ede0 secure :-1 s=READY pgs=5700 cs=0 l=1 rev1=1 rx=0x7f21cc0433a0 tx=0x7f21e40a6df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:24.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ede0 secure :-1 s=READY pgs=5700 cs=0 l=1 rev1=1 rx=0x7f21cc0433a0 tx=0x7f21e40a6df0).stop 2021-03-09T09:39:24.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ede0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:24.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ede0 unknown :-1 s=CLOSED pgs=5700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:24.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:24.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc050080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:24.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:24.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054390 con 0x7f21cc005e30 2021-03-09T09:39:24.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030240 con 0x7f21c4003cb0 2021-03-09T09:39:24.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057960 con 0x7f21c4008860 2021-03-09T09:39:24.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:24.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:24.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:24.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:24.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc050080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:24.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:24.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:24.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:24.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:24.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e5e0 con 0x7f21c4008860 2021-03-09T09:39:24.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f370 con 0x7f21c4008860 2021-03-09T09:39:24.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc050080 secure :-1 s=READY pgs=5711 cs=0 l=1 rev1=1 rx=0x7f21cc00b3b0 tx=0x7f21e4009230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:24.121 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:24.108+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:24.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c4008860 2021-03-09T09:39:24.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40310b0 con 0x7f21c4008860 2021-03-09T09:39:24.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c4008860 2021-03-09T09:39:24.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.116+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:24.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:24.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:24.116+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:29.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:29.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:29.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc050080 secure :-1 s=READY pgs=5711 cs=0 l=1 rev1=1 rx=0x7f21cc00b3b0 tx=0x7f21e4009230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:29.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc050080 secure :-1 s=READY pgs=5711 cs=0 l=1 rev1=1 rx=0x7f21cc00b3b0 tx=0x7f21e4009230).stop 2021-03-09T09:39:29.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc050080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:29.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc050080 unknown :-1 s=CLOSED pgs=5711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:29.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:29.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:29.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:29.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02db10 con 0x7f21cc005e30 2021-03-09T09:39:29.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014e80 con 0x7f21c4003cb0 2021-03-09T09:39:29.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1f0 con 0x7f21ec0a7f50 2021-03-09T09:39:29.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:29.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:29.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:29.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:29.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:29.120 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19160 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:29.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:29.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:29.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:29.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:29.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f510 con 0x7f21ec0a7f50 2021-03-09T09:39:29.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.112+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b0e0 con 0x7f21ec0a7f50 2021-03-09T09:39:29.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ed90 secure :-1 s=READY pgs=5724 cs=0 l=1 rev1=1 rx=0x7f21dc0333f0 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:29.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21ec0a7f50 2021-03-09T09:39:29.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089c30 con 0x7f21ec0a7f50 2021-03-09T09:39:29.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc08c000 con 0x7f21ec0a7f50 2021-03-09T09:39:29.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.124+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:29.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:29.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:29.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:34.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:34.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:34.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ed90 secure :-1 s=READY pgs=5724 cs=0 l=1 rev1=1 rx=0x7f21dc0333f0 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:34.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ed90 secure :-1 s=READY pgs=5724 cs=0 l=1 rev1=1 rx=0x7f21dc0333f0 tx=0x7f21dc009810).stop 2021-03-09T09:39:34.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc03ed90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:34.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc03ed90 unknown :-1 s=CLOSED pgs=5724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:34.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:34.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:34.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:34.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b280 con 0x7f21cc005e30 2021-03-09T09:39:34.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04adc0 con 0x7f21c4003cb0 2021-03-09T09:39:34.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014e80 con 0x7f21c4008860 2021-03-09T09:39:34.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:34.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:34.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:34.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:34.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:34.125 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19200 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:34.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:34.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:34.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:34.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:34.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02db10 con 0x7f21c4008860 2021-03-09T09:39:34.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f510 con 0x7f21c4008860 2021-03-09T09:39:34.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c70 secure :-1 s=READY pgs=5732 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc062ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:34.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0505a0 con 0x7f21c4008860 2021-03-09T09:39:34.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4008860 2021-03-09T09:39:34.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc050390 con 0x7f21c4008860 2021-03-09T09:39:34.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.124+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:34.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:34.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:34.124+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:39.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc022c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:39:39.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc022c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:39.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c70 secure :-1 s=READY pgs=5732 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc062ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:39.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c70 secure :-1 s=READY pgs=5732 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc062ec0).stop 2021-03-09T09:39:39.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc022c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:39.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c70 unknown :-1 s=CLOSED pgs=5732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:39.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:39.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:39.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:39.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056f00 con 0x7f21cc005e30 2021-03-09T09:39:39.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014e80 con 0x7f21c4003cb0 2021-03-09T09:39:39.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023160 con 0x7f21ec0a7f50 2021-03-09T09:39:39.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:39.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:39.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:39.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:39.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:39.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:39.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:39.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:39.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:39.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04adc0 con 0x7f21ec0a7f50 2021-03-09T09:39:39.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e5e0 con 0x7f21ec0a7f50 2021-03-09T09:39:39.130 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19235 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:39.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 secure :-1 s=READY pgs=5745 cs=0 l=1 rev1=1 rx=0x7f21e407d4e0 tx=0x7f21e4036d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:39.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d320 con 0x7f21ec0a7f50 2021-03-09T09:39:39.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a6e60 con 0x7f21ec0a7f50 2021-03-09T09:39:39.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21ec0a7f50 2021-03-09T09:39:39.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.128+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:39.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.128+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:39.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:39.128+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:44.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:44.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:44.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 secure :-1 s=READY pgs=5745 cs=0 l=1 rev1=1 rx=0x7f21e407d4e0 tx=0x7f21e4036d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:44.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 secure :-1 s=READY pgs=5745 cs=0 l=1 rev1=1 rx=0x7f21e407d4e0 tx=0x7f21e4036d30).stop 2021-03-09T09:39:44.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:44.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 unknown :-1 s=CLOSED pgs=5745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:44.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:44.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:44.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:44.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc014e80 con 0x7f21cc005e30 2021-03-09T09:39:44.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057c00 con 0x7f21c4003cb0 2021-03-09T09:39:44.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cde0 con 0x7f21c4008860 2021-03-09T09:39:44.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:44.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:44.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:44.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:44.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:44.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:44.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:44.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:44.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:44.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b280 con 0x7f21c4008860 2021-03-09T09:39:44.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04adc0 con 0x7f21c4008860 2021-03-09T09:39:44.134 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19271 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:44.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c10 secure :-1 s=READY pgs=5754 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4015be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:44.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d360 con 0x7f21c4008860 2021-03-09T09:39:44.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d360 con 0x7f21c4008860 2021-03-09T09:39:44.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21c4008860 2021-03-09T09:39:44.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.132+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:44.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.132+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:44.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:44.132+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:49.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc022c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:49.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc022c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:49.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c10 secure :-1 s=READY pgs=5754 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4015be0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:49.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c10 secure :-1 s=READY pgs=5754 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4015be0).stop 2021-03-09T09:39:49.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc022c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc022c10 unknown :-1 s=CLOSED pgs=5754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:49.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c70 con 0x7f21cc005e30 2021-03-09T09:39:49.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21c4003cb0 2021-03-09T09:39:49.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e3d0 con 0x7f21ec0a7f50 2021-03-09T09:39:49.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:49.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:49.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:49.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:49.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:49.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:49.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:49.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:49.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:49.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e510 con 0x7f21ec0a7f50 2021-03-09T09:39:49.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02db10 con 0x7f21ec0a7f50 2021-03-09T09:39:49.138 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19309 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:49.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 rx=0x7f21dc063c90 tx=0x7f21dc062d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:49.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d60 con 0x7f21ec0a7f50 2021-03-09T09:39:49.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21ec0a7f50 2021-03-09T09:39:49.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21ec0a7f50 2021-03-09T09:39:49.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.132+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:49.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.136+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:49.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:49.136+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:51.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:51.461 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:39:51.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900001760) for more than 900000000 us, fault. 2021-03-09T09:39:52.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e001f730).stop 2021-03-09T09:39:52.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21cc083d30 2021-03-09T09:39:52.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T09:39:52.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:52.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=CLOSED pgs=3690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:52.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:52.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 0x7f21cc0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:52.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 0x7f21cc0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:52.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:51.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 0x7f21cc0048d0 secure :-1 s=READY pgs=5498 cs=0 l=1 rev1=1 rx=0x7f21e0005450 tx=0x7f21e001f730).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:54.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:54.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:54.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 rx=0x7f21dc063c90 tx=0x7f21dc062d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:54.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 rx=0x7f21dc063c90 tx=0x7f21dc062d90).stop 2021-03-09T09:39:54.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc050080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:54.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc050080 unknown :-1 s=CLOSED pgs=5763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:54.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:54.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:54.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:54.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f240 con 0x7f21cc005e30 2021-03-09T09:39:54.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004dc0 con 0x7f21c4003cb0 2021-03-09T09:39:54.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c70 con 0x7f21cc083d30 2021-03-09T09:39:54.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:54.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:54.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:54.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:54.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:54.140 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19345 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:54.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:54.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:54.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:54.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:54.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e380 con 0x7f21cc083d30 2021-03-09T09:39:54.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.131+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e510 con 0x7f21cc083d30 2021-03-09T09:39:54.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=5777 cs=0 l=1 rev1=1 rx=0x7f21dc089d30 tx=0x7f21dc0b31d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:54.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3500 con 0x7f21cc083d30 2021-03-09T09:39:54.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0897b0 con 0x7f21cc083d30 2021-03-09T09:39:54.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc083d30 2021-03-09T09:39:54.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.139+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:54.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:54.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:54.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:39:59.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:59.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:59.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=5777 cs=0 l=1 rev1=1 rx=0x7f21dc089d30 tx=0x7f21dc0b31d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:59.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 secure :-1 s=READY pgs=5777 cs=0 l=1 rev1=1 rx=0x7f21dc089d30 tx=0x7f21dc0b31d0).stop 2021-03-09T09:39:59.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc0570d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc0570d0 unknown :-1 s=CLOSED pgs=5777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c70 con 0x7f21cc005e30 2021-03-09T09:39:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091690 con 0x7f21c4003cb0 2021-03-09T09:39:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21ec0a7f50 2021-03-09T09:39:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:39:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:59.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:39:59.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:59.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:59.144 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19383 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:39:59.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:59.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:59.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:59.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:59.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023100 con 0x7f21ec0a7f50 2021-03-09T09:39:59.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b280 con 0x7f21ec0a7f50 2021-03-09T09:39:59.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.135+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022c10 secure :-1 s=READY pgs=5785 cs=0 l=1 rev1=1 rx=0x7f21e4036f60 tx=0x7f21e4059070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:59.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4015d70 con 0x7f21ec0a7f50 2021-03-09T09:39:59.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d150 con 0x7f21ec0a7f50 2021-03-09T09:39:59.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4015d70 con 0x7f21ec0a7f50 2021-03-09T09:39:59.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.143+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:39:59.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:39:59.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:39:59.143+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:04.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:04.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:04.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022c10 secure :-1 s=READY pgs=5785 cs=0 l=1 rev1=1 rx=0x7f21e4036f60 tx=0x7f21e4059070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:04.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022c10 secure :-1 s=READY pgs=5785 cs=0 l=1 rev1=1 rx=0x7f21e4036f60 tx=0x7f21e4059070).stop 2021-03-09T09:40:04.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:04.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022c10 unknown :-1 s=CLOSED pgs=5785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:04.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:04.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:04.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:04.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21cc005e30 2021-03-09T09:40:04.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c70 con 0x7f21c4003cb0 2021-03-09T09:40:04.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc083d30 2021-03-09T09:40:04.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:04.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:04.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:04.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.139+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:04.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:04.147 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19419 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057090 con 0x7f21cc083d30 2021-03-09T09:40:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023100 con 0x7f21cc083d30 2021-03-09T09:40:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089ce0 secure :-1 s=READY pgs=5797 cs=0 l=1 rev1=1 rx=0x7f21cc0451a0 tx=0x7f21e4015b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:04.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21cc083d30 2021-03-09T09:40:04.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009e30 con 0x7f21cc083d30 2021-03-09T09:40:04.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21cc083d30 2021-03-09T09:40:04.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.151+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:04.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.155+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:04.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:04.155+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:09.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:09.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:09.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089ce0 secure :-1 s=READY pgs=5797 cs=0 l=1 rev1=1 rx=0x7f21cc0451a0 tx=0x7f21e4015b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:09.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089ce0 secure :-1 s=READY pgs=5797 cs=0 l=1 rev1=1 rx=0x7f21cc0451a0 tx=0x7f21e4015b30).stop 2021-03-09T09:40:09.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:09.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089ce0 unknown :-1 s=CLOSED pgs=5797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:09.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:09.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:09.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:09.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21cc005e30 2021-03-09T09:40:09.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c70 con 0x7f21c4003cb0 2021-03-09T09:40:09.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022c10 con 0x7f21ec0a7f50 2021-03-09T09:40:09.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:09.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:09.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:09.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:09.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.143+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:09.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:09.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:09.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:09.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:09.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022d50 con 0x7f21ec0a7f50 2021-03-09T09:40:09.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e380 con 0x7f21ec0a7f50 2021-03-09T09:40:09.153 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19457 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5806 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0abc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21ec0a7f50 2021-03-09T09:40:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018380 con 0x7f21ec0a7f50 2021-03-09T09:40:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21ec0a7f50 2021-03-09T09:40:09.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.151+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:09.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.151+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:09.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:09.151+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:14.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:14.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:14.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5806 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0abc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:14.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5806 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0abc90).stop 2021-03-09T09:40:14.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:14.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=CLOSED pgs=5806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:14.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:14.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:14.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:14.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:14.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:14.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c70 con 0x7f21cc005e30 2021-03-09T09:40:14.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0082d0 con 0x7f21c4003cb0 2021-03-09T09:40:14.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e520 con 0x7f21cc083d30 2021-03-09T09:40:14.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:14.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:14.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.147+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:14.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:14.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:14.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:14.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:14.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc083d30 2021-03-09T09:40:14.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022d50 con 0x7f21cc083d30 2021-03-09T09:40:14.162 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19492 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:14.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089cc0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0646c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:14.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc028210 con 0x7f21cc083d30 2021-03-09T09:40:14.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:40:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc083d30 2021-03-09T09:40:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.155+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.159+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:14.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:14.159+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:19.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:19.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:19.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089cc0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0646c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:19.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089cc0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0646c0).stop 2021-03-09T09:40:19.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:19.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089cc0 unknown :-1 s=CLOSED pgs=5817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:19.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:19.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:19.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:19.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e520 con 0x7f21cc005e30 2021-03-09T09:40:19.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026de0 con 0x7f21c4003cb0 2021-03-09T09:40:19.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057710 con 0x7f21ec0a7f50 2021-03-09T09:40:19.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:19.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:19.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:19.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:19.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.151+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:19.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:19.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:19.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:19.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:19.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022d50 con 0x7f21ec0a7f50 2021-03-09T09:40:19.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:40:19.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5827 cs=0 l=1 rev1=1 rx=0x7f21dc064a40 tx=0x7f21dc0688f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:19.162 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19531 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:19.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc091e50 con 0x7f21ec0a7f50 2021-03-09T09:40:19.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:40:19.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.159+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21ec0a7f50 2021-03-09T09:40:19.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.159+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:19.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.163+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:19.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:19.163+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:21.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:21.529 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:24.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:24.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:24.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5827 cs=0 l=1 rev1=1 rx=0x7f21dc064a40 tx=0x7f21dc0688f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:24.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5827 cs=0 l=1 rev1=1 rx=0x7f21dc064a40 tx=0x7f21dc0688f0).stop 2021-03-09T09:40:24.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:24.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=CLOSED pgs=5827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:24.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:24.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:24.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:24.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:40:24.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0091c0 con 0x7f21c4003cb0 2021-03-09T09:40:24.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009300 con 0x7f21cc083d30 2021-03-09T09:40:24.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:24.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:24.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:24.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:24.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:24.163 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:24.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:24.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:24.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:24.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:24.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089c60 con 0x7f21cc083d30 2021-03-09T09:40:24.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022d50 con 0x7f21cc083d30 2021-03-09T09:40:24.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022ee0 secure :-1 s=READY pgs=5840 cs=0 l=1 rev1=1 rx=0x7f21dc0288a0 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:24.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.163+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc083d30 2021-03-09T09:40:24.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc083d30 2021-03-09T09:40:24.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc077fc0 con 0x7f21cc083d30 2021-03-09T09:40:24.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.171+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:24.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.171+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:24.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:24.171+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:29.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:29.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:29.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022ee0 secure :-1 s=READY pgs=5840 cs=0 l=1 rev1=1 rx=0x7f21dc0288a0 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:29.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022ee0 secure :-1 s=READY pgs=5840 cs=0 l=1 rev1=1 rx=0x7f21dc0288a0 tx=0x7f21dc07ff90).stop 2021-03-09T09:40:29.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc022ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:29.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc022ee0 unknown :-1 s=CLOSED pgs=5840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:29.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:29.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:29.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:29.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037910 con 0x7f21cc005e30 2021-03-09T09:40:29.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057710 con 0x7f21c4003cb0 2021-03-09T09:40:29.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026dc0 con 0x7f21ec0a7f50 2021-03-09T09:40:29.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:29.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:29.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:29.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:29.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.159+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:29.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:29.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:29.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:29.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:29.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d830 con 0x7f21ec0a7f50 2021-03-09T09:40:29.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089c60 con 0x7f21ec0a7f50 2021-03-09T09:40:29.169 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19605 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:29.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5848 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc085280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:29.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21ec0a7f50 2021-03-09T09:40:29.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21ec0a7f50 2021-03-09T09:40:29.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc068d80 con 0x7f21ec0a7f50 2021-03-09T09:40:29.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.171+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:29.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.171+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:29.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:29.171+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:34.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:34.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:34.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5848 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc085280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:34.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 secure :-1 s=READY pgs=5848 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc085280).stop 2021-03-09T09:40:34.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc057220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:34.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc057220 unknown :-1 s=CLOSED pgs=5848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:34.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:34.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:34.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:34.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:40:34.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a070 con 0x7f21c4003cb0 2021-03-09T09:40:34.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08db10 con 0x7f21cc083d30 2021-03-09T09:40:34.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:34.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:34.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:34.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:34.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.163+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:34.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:34.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:34.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:34.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:34.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08dc50 con 0x7f21cc083d30 2021-03-09T09:40:34.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d830 con 0x7f21cc083d30 2021-03-09T09:40:34.173 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19640 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:34.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089b80 secure :-1 s=READY pgs=5859 cs=0 l=1 rev1=1 rx=0x7f21dc0292e0 tx=0x7f21dc029120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:34.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc083d30 2021-03-09T09:40:34.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc083d30 2021-03-09T09:40:34.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.175+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc083d30 2021-03-09T09:40:34.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.175+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:34.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.175+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:34.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:34.175+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:39.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:39.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:39.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089b80 secure :-1 s=READY pgs=5859 cs=0 l=1 rev1=1 rx=0x7f21dc0292e0 tx=0x7f21dc029120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:39.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089b80 secure :-1 s=READY pgs=5859 cs=0 l=1 rev1=1 rx=0x7f21dc0292e0 tx=0x7f21dc029120).stop 2021-03-09T09:40:39.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc089b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:39.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc089b80 unknown :-1 s=CLOSED pgs=5859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:39.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:39.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:39.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:39.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023210 con 0x7f21cc005e30 2021-03-09T09:40:39.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08db10 con 0x7f21c4003cb0 2021-03-09T09:40:39.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dc50 con 0x7f21ec0a7f50 2021-03-09T09:40:39.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:39.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:39.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:39.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:39.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:39.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:39.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:39.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:39.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:39.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026a80 con 0x7f21ec0a7f50 2021-03-09T09:40:39.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.167+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057090 con 0x7f21ec0a7f50 2021-03-09T09:40:39.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5870 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4009470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:39.184 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:39.171+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19679 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:39.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:40:39.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:40:39.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.175+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21ec0a7f50 2021-03-09T09:40:39.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.175+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:39.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.175+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:39.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:39.175+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:44.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:44.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:44.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5870 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4009470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:44.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5870 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4009470).stop 2021-03-09T09:40:44.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:44.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=CLOSED pgs=5870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:44.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:44.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:44.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:44.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:40:44.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089b30 con 0x7f21c4003cb0 2021-03-09T09:40:44.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089c70 con 0x7f21cc083d30 2021-03-09T09:40:44.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:44.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:44.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:44.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:44.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:44.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:44.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:44.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:44.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:44.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.171+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089db0 con 0x7f21cc083d30 2021-03-09T09:40:44.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.175+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d830 con 0x7f21cc083d30 2021-03-09T09:40:44.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.175+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5880 cs=0 l=1 rev1=1 rx=0x7f21dc016d60 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:44.181 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:44.175+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19714 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:44.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.175+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018b70 con 0x7f21cc083d30 2021-03-09T09:40:44.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.175+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018940 con 0x7f21cc083d30 2021-03-09T09:40:44.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.179+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021d70 con 0x7f21cc083d30 2021-03-09T09:40:44.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.179+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:44.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.179+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:44.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:44.179+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:49.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:49.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:49.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5880 cs=0 l=1 rev1=1 rx=0x7f21dc016d60 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:49.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5880 cs=0 l=1 rev1=1 rx=0x7f21dc016d60 tx=0x7f21dc077e50).stop 2021-03-09T09:40:49.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:49.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=CLOSED pgs=5880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:49.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:49.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:49.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:49.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:49.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:49.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a0e0 con 0x7f21cc005e30 2021-03-09T09:40:49.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094370 con 0x7f21c4003cb0 2021-03-09T09:40:49.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023210 con 0x7f21ec0a7f50 2021-03-09T09:40:49.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:49.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:49.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.174+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:49.183 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19753 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:49.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:49.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:49.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:49.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:49.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026880 con 0x7f21ec0a7f50 2021-03-09T09:40:49.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089db0 con 0x7f21ec0a7f50 2021-03-09T09:40:49.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5893 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc018820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:49.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.182+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21ec0a7f50 2021-03-09T09:40:49.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.182+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21ec0a7f50 2021-03-09T09:40:49.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021690 con 0x7f21ec0a7f50 2021-03-09T09:40:49.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.186+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:49.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.186+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:49.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:49.186+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:51.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:51.582 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:40:54.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:54.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:54.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5893 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc018820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:54.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5893 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc018820).stop 2021-03-09T09:40:54.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:54.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=CLOSED pgs=5893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:54.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:54.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:54.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:54.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03cf20 con 0x7f21cc005e30 2021-03-09T09:40:54.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015570 con 0x7f21c4003cb0 2021-03-09T09:40:54.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc083d30 2021-03-09T09:40:54.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:54.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:54.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:54.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:54.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.178+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:54.192 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19788 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:54.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:54.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:54.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:54.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:54.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052e60 con 0x7f21cc083d30 2021-03-09T09:40:54.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026880 con 0x7f21cc083d30 2021-03-09T09:40:54.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5905 cs=0 l=1 rev1=1 rx=0x7f21cc027990 tx=0x7f21dc0205c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:54.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08c9c0 con 0x7f21cc083d30 2021-03-09T09:40:54.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08c9c0 con 0x7f21cc083d30 2021-03-09T09:40:54.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.190+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06bdc0 con 0x7f21cc083d30 2021-03-09T09:40:54.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.190+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:54.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.190+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:54.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:54.190+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:40:59.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:59.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:59.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5905 cs=0 l=1 rev1=1 rx=0x7f21cc027990 tx=0x7f21dc0205c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:59.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5905 cs=0 l=1 rev1=1 rx=0x7f21cc027990 tx=0x7f21dc0205c0).stop 2021-03-09T09:40:59.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:59.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=CLOSED pgs=5905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:59.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:59.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:59.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:59.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d0c0 con 0x7f21cc005e30 2021-03-09T09:40:59.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015570 con 0x7f21c4003cb0 2021-03-09T09:40:59.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21ec0a7f50 2021-03-09T09:40:59.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:40:59.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:59.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:40:59.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:59.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.182+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:59.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:59.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:59.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:59.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:59.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026880 con 0x7f21ec0a7f50 2021-03-09T09:40:59.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026a80 con 0x7f21ec0a7f50 2021-03-09T09:40:59.193 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19827 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:40:59.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5912 cs=0 l=1 rev1=1 rx=0x7f21cc02f4b0 tx=0x7f21e408f140).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:59.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4028b10 con 0x7f21ec0a7f50 2021-03-09T09:40:59.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:40:59.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21ec0a7f50 2021-03-09T09:40:59.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.194+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:40:59.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.194+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:40:59.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:40:59.194+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:04.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:04.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:04.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5912 cs=0 l=1 rev1=1 rx=0x7f21cc02f4b0 tx=0x7f21e408f140).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:04.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5912 cs=0 l=1 rev1=1 rx=0x7f21cc02f4b0 tx=0x7f21e408f140).stop 2021-03-09T09:41:04.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:04.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=CLOSED pgs=5912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:04.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:04.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:04.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:04.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d260 con 0x7f21cc005e30 2021-03-09T09:41:04.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015570 con 0x7f21c4003cb0 2021-03-09T09:41:04.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055490 con 0x7f21cc083d30 2021-03-09T09:41:04.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:04.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:04.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:04.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:04.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.186+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:04.196 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19862 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:04.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:04.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:04.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:04.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:04.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc083d30 2021-03-09T09:41:04.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052e60 con 0x7f21cc083d30 2021-03-09T09:41:04.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5925 cs=0 l=1 rev1=1 rx=0x7f21dc08cb80 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:04.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018b70 con 0x7f21cc083d30 2021-03-09T09:41:04.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018b70 con 0x7f21cc083d30 2021-03-09T09:41:04.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.198+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc083d30 2021-03-09T09:41:04.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.198+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:04.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.198+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:04.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:04.198+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:09.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:09.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:09.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5925 cs=0 l=1 rev1=1 rx=0x7f21dc08cb80 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:09.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5925 cs=0 l=1 rev1=1 rx=0x7f21dc08cb80 tx=0x7f21dc010d20).stop 2021-03-09T09:41:09.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:09.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=CLOSED pgs=5925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:09.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:09.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:09.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:09.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:41:09.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052ff0 con 0x7f21c4003cb0 2021-03-09T09:41:09.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053130 con 0x7f21ec0a7f50 2021-03-09T09:41:09.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:09.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:09.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:09.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:09.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.190+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:09.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:09.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:09.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:09.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:09.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03cf20 con 0x7f21ec0a7f50 2021-03-09T09:41:09.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052e60 con 0x7f21ec0a7f50 2021-03-09T09:41:09.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5934 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0623c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:09.202 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19901 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:09.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21ec0a7f50 2021-03-09T09:41:09.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21ec0a7f50 2021-03-09T09:41:09.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.198+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018d30 con 0x7f21ec0a7f50 2021-03-09T09:41:09.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.198+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:09.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.202+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:09.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:09.202+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:14.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:14.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:14.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5934 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0623c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:14.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5934 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0623c0).stop 2021-03-09T09:41:14.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:14.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=CLOSED pgs=5934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:14.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:14.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:14.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:14.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d250 con 0x7f21cc005e30 2021-03-09T09:41:14.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0304a0 con 0x7f21c4003cb0 2021-03-09T09:41:14.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc083d30 2021-03-09T09:41:14.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:14.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:14.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:14.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:14.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.194+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:14.204 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19936 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:14.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:14.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:14.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:14.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:14.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052e60 con 0x7f21cc083d30 2021-03-09T09:41:14.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026880 con 0x7f21cc083d30 2021-03-09T09:41:14.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.198+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5947 cs=0 l=1 rev1=1 rx=0x7f21e4053fd0 tx=0x7f21e4015f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:14.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.202+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5d20 con 0x7f21cc083d30 2021-03-09T09:41:14.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.202+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5d20 con 0x7f21cc083d30 2021-03-09T09:41:14.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.206+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4015c00 con 0x7f21cc083d30 2021-03-09T09:41:14.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.206+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:14.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.206+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:14.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:14.206+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:19.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:19.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:19.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5947 cs=0 l=1 rev1=1 rx=0x7f21e4053fd0 tx=0x7f21e4015f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:19.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5947 cs=0 l=1 rev1=1 rx=0x7f21e4053fd0 tx=0x7f21e4015f00).stop 2021-03-09T09:41:19.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:19.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=CLOSED pgs=5947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:19.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:19.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:19.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:19.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f560 con 0x7f21cc005e30 2021-03-09T09:41:19.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0304a0 con 0x7f21c4003cb0 2021-03-09T09:41:19.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052ff0 con 0x7f21ec0a7f50 2021-03-09T09:41:19.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:19.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:19.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:19.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:19.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.198+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:19.207 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19975 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:19.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:19.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:19.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:19.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:19.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053160 con 0x7f21ec0a7f50 2021-03-09T09:41:19.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052e60 con 0x7f21ec0a7f50 2021-03-09T09:41:19.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.202+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5956 cs=0 l=1 rev1=1 rx=0x7f21e4080050 tx=0x7f21e40b03c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:19.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a4c90 con 0x7f21ec0a7f50 2021-03-09T09:41:19.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a4df0 con 0x7f21ec0a7f50 2021-03-09T09:41:19.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40179f0 con 0x7f21ec0a7f50 2021-03-09T09:41:19.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.210+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:19.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.210+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:19.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:19.210+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:21.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:21.629 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:24.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:24.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:24.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5956 cs=0 l=1 rev1=1 rx=0x7f21e4080050 tx=0x7f21e40b03c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:24.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 secure :-1 s=READY pgs=5956 cs=0 l=1 rev1=1 rx=0x7f21e4080050 tx=0x7f21e40b03c0).stop 2021-03-09T09:41:24.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:24.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc022d20 unknown :-1 s=CLOSED pgs=5956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:24.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:24.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:24.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:24.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f560 con 0x7f21cc005e30 2021-03-09T09:41:24.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b570 con 0x7f21c4003cb0 2021-03-09T09:41:24.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053000 con 0x7f21cc083d30 2021-03-09T09:41:24.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:24.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:24.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:24.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:24.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.202+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:24.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:24.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:24.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc01abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:24.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc01abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:24.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0304a0 con 0x7f21cc083d30 2021-03-09T09:41:24.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03cf20 con 0x7f21cc083d30 2021-03-09T09:41:24.213 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20010 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:24.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5966 cs=0 l=1 rev1=1 rx=0x7f21cc02ea10 tx=0x7f21dc083f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:24.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.206+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05b960 con 0x7f21cc083d30 2021-03-09T09:41:24.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05b960 con 0x7f21cc083d30 2021-03-09T09:41:24.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc083d30 2021-03-09T09:41:24.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.214+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:24.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.214+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:24.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:24.214+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:29.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:29.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:29.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5966 cs=0 l=1 rev1=1 rx=0x7f21cc02ea10 tx=0x7f21dc083f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:29.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5966 cs=0 l=1 rev1=1 rx=0x7f21cc02ea10 tx=0x7f21dc083f20).stop 2021-03-09T09:41:29.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:29.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=CLOSED pgs=5966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:29.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:29.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:29.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:29.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f560 con 0x7f21cc005e30 2021-03-09T09:41:29.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045010 con 0x7f21c4003cb0 2021-03-09T09:41:29.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21ec0a7f50 2021-03-09T09:41:29.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:29.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:29.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:29.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:29.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:29.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:29.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:29.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:29.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:29.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.206+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03d0c0 con 0x7f21ec0a7f50 2021-03-09T09:41:29.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.210+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053160 con 0x7f21ec0a7f50 2021-03-09T09:41:29.216 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:29.210+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20049 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:29.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.210+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ab10 secure :-1 s=READY pgs=5976 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4024000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:29.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f250 con 0x7f21ec0a7f50 2021-03-09T09:41:29.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40092a0 con 0x7f21ec0a7f50 2021-03-09T09:41:29.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21ec0a7f50 2021-03-09T09:41:29.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.214+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:29.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.214+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:29.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:29.214+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:34.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:34.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:34.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ab10 secure :-1 s=READY pgs=5976 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4024000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:34.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ab10 secure :-1 s=READY pgs=5976 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4024000).stop 2021-03-09T09:41:34.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01ab10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:34.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01ab10 unknown :-1 s=CLOSED pgs=5976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:34.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:34.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:34.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:34.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f560 con 0x7f21cc005e30 2021-03-09T09:41:34.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T09:41:34.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b000 con 0x7f21cc083d30 2021-03-09T09:41:34.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:34.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:34.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:34.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:34.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:34.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:34.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:34.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:34.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:34.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b170 con 0x7f21cc083d30 2021-03-09T09:41:34.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.210+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0304a0 con 0x7f21cc083d30 2021-03-09T09:41:34.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.214+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5989 cs=0 l=1 rev1=1 rx=0x7f21dc050d50 tx=0x7f21dc0adf50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:34.221 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:34.214+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20084 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:34.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc083d30 2021-03-09T09:41:34.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc083d30 2021-03-09T09:41:34.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.222+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc083d30 2021-03-09T09:41:34.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.222+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:34.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.222+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:34.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:34.222+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:39.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:41:39.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:39.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5989 cs=0 l=1 rev1=1 rx=0x7f21dc050d50 tx=0x7f21dc0adf50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:39.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 secure :-1 s=READY pgs=5989 cs=0 l=1 rev1=1 rx=0x7f21dc050d50 tx=0x7f21dc0adf50).stop 2021-03-09T09:41:39.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc057090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:39.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc057090 unknown :-1 s=CLOSED pgs=5989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:39.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:39.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:39.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:39.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0304a0 con 0x7f21cc005e30 2021-03-09T09:41:39.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0267e0 con 0x7f21c4003cb0 2021-03-09T09:41:39.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae60 con 0x7f21ec0a7f50 2021-03-09T09:41:39.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:39.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:39.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:39.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:39.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.214+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:39.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:39.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:39.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:39.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:39.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01afa0 con 0x7f21ec0a7f50 2021-03-09T09:41:39.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b170 con 0x7f21ec0a7f50 2021-03-09T09:41:39.224 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20123 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:39.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.218+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a970 secure :-1 s=READY pgs=5998 cs=0 l=1 rev1=1 rx=0x7f21dc00a810 tx=0x7f21dc0adee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:39.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.222+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21ec0a7f50 2021-03-09T09:41:39.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.222+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21ec0a7f50 2021-03-09T09:41:39.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.222+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21ec0a7f50 2021-03-09T09:41:39.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.222+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:39.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.226+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:39.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:39.226+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:44.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:44.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a970 secure :-1 s=READY pgs=5998 cs=0 l=1 rev1=1 rx=0x7f21dc00a810 tx=0x7f21dc0adee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a970 secure :-1 s=READY pgs=5998 cs=0 l=1 rev1=1 rx=0x7f21dc00a810 tx=0x7f21dc0adee0).stop 2021-03-09T09:41:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 msgr2=0x7f21cc01a970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21ec0a7f50 0x7f21cc01a970 unknown :-1 s=CLOSED pgs=5998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0304a0 con 0x7f21cc005e30 2021-03-09T09:41:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T09:41:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0422e0 con 0x7f21cc083d30 2021-03-09T09:41:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:44.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:44.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:44.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:44.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:44.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:44.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:44.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042420 con 0x7f21cc083d30 2021-03-09T09:41:44.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01afa0 con 0x7f21cc083d30 2021-03-09T09:41:44.228 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20158 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:44.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02d550 secure :-1 s=READY pgs=6008 cs=0 l=1 rev1=1 rx=0x7f21dc028b00 tx=0x7f21dc083cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:44.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc083d30 2021-03-09T09:41:44.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21cc083d30 2021-03-09T09:41:44.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc083d30 2021-03-09T09:41:44.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.229+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:44.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.229+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:44.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:44.229+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:49.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:49.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:49.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02d550 secure :-1 s=READY pgs=6008 cs=0 l=1 rev1=1 rx=0x7f21dc028b00 tx=0x7f21dc083cc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:49.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02d550 secure :-1 s=READY pgs=6008 cs=0 l=1 rev1=1 rx=0x7f21dc028b00 tx=0x7f21dc083cc0).stop 2021-03-09T09:41:49.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 msgr2=0x7f21cc02d550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:49.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc083d30 0x7f21cc02d550 unknown :-1 s=CLOSED pgs=6008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:49.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:49.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:49.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc042a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:49.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0422c0 con 0x7f21cc005e30 2021-03-09T09:41:49.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0304a0 con 0x7f21c4003cb0 2021-03-09T09:41:49.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01acc0 con 0x7f21cc0425b0 2021-03-09T09:41:49.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:49.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:49.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:49.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:49.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc042a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:49.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:49.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:49.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:49.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:49.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:41:49.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042420 con 0x7f21cc0425b0 2021-03-09T09:41:49.232 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:49.225+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20197 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:49.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.225+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc042a30 secure :-1 s=READY pgs=6019 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07fd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:49.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc0425b0 2021-03-09T09:41:49.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031fc0 con 0x7f21cc0425b0 2021-03-09T09:41:49.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc0425b0 2021-03-09T09:41:49.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.229+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:49.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.233+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:49.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:49.233+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:51.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:51.668 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:41:54.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc042a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:54.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc042a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:54.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc042a30 secure :-1 s=READY pgs=6019 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07fd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:54.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc042a30 secure :-1 s=READY pgs=6019 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc07fd90).stop 2021-03-09T09:41:54.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc042a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:54.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc042a30 unknown :-1 s=CLOSED pgs=6019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:54.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:54.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:54.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:54.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:41:54.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d070 con 0x7f21c4003cb0 2021-03-09T09:41:54.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01acc0 con 0x7f21cc042f20 2021-03-09T09:41:54.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:54.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:54.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:54.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:54.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:54.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:54.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:54.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:54.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:54.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0304a0 con 0x7f21cc042f20 2021-03-09T09:41:54.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03d0c0 con 0x7f21cc042f20 2021-03-09T09:41:54.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.225+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d210 secure :-1 s=READY pgs=6030 cs=0 l=1 rev1=1 rx=0x7f21e4024350 tx=0x7f21e4024260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:54.236 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:54.229+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20232 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:54.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407e260 con 0x7f21cc042f20 2021-03-09T09:41:54.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0070 con 0x7f21cc042f20 2021-03-09T09:41:54.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027d20 con 0x7f21cc042f20 2021-03-09T09:41:54.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.233+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:54.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.237+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:54.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:54.237+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:41:59.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:59.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:59.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d210 secure :-1 s=READY pgs=6030 cs=0 l=1 rev1=1 rx=0x7f21e4024350 tx=0x7f21e4024260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:59.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d210 secure :-1 s=READY pgs=6030 cs=0 l=1 rev1=1 rx=0x7f21e4024350 tx=0x7f21e4024260).stop 2021-03-09T09:41:59.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:59.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d210 unknown :-1 s=CLOSED pgs=6030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:59.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:59.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:59.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:59.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21cc005e30 2021-03-09T09:41:59.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21c4003cb0 2021-03-09T09:41:59.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b050 con 0x7f21cc0425b0 2021-03-09T09:41:59.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:41:59.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:59.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:41:59.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:59.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.229+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:59.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:59.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:59.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:59.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:59.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03d0c0 con 0x7f21cc0425b0 2021-03-09T09:41:59.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:41:59.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d700 secure :-1 s=READY pgs=6040 cs=0 l=1 rev1=1 rx=0x7f21dc085340 tx=0x7f21dc083d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:59.241 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20271 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:41:59.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc0425b0 2021-03-09T09:41:59.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc0425b0 2021-03-09T09:41:59.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc0425b0 2021-03-09T09:41:59.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.241+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:41:59.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.241+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:41:59.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:41:59.241+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:04.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:04.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:04.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d700 secure :-1 s=READY pgs=6040 cs=0 l=1 rev1=1 rx=0x7f21dc085340 tx=0x7f21dc083d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:04.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d700 secure :-1 s=READY pgs=6040 cs=0 l=1 rev1=1 rx=0x7f21dc085340 tx=0x7f21dc083d30).stop 2021-03-09T09:42:04.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02d700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:04.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d700 unknown :-1 s=CLOSED pgs=6040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:04.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:04.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:04.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:04.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f6b0 con 0x7f21cc005e30 2021-03-09T09:42:04.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21c4003cb0 2021-03-09T09:42:04.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21cc042f20 2021-03-09T09:42:04.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:04.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:04.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:04.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:04.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.233+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:04.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:04.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:04.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:04.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:04.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:42:04.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03d0c0 con 0x7f21cc042f20 2021-03-09T09:42:04.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6051 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0adf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:04.244 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:04.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc042f20 2021-03-09T09:42:04.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028650 con 0x7f21cc042f20 2021-03-09T09:42:04.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc09a070 con 0x7f21cc042f20 2021-03-09T09:42:04.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.241+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:04.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.245+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:04.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:04.245+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:09.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:09.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:09.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6051 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0adf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:09.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6051 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0adf70).stop 2021-03-09T09:42:09.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:09.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:09.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:09.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:09.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:09.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:09.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0350b0 con 0x7f21c4003cb0 2021-03-09T09:42:09.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21cc0425b0 2021-03-09T09:42:09.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:09.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:09.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:09.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:09.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:09.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:09.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:09.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:09.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:09.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03d0c0 con 0x7f21cc0425b0 2021-03-09T09:42:09.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc0425b0 2021-03-09T09:42:09.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6062 cs=0 l=1 rev1=1 rx=0x7f21dc0ac380 tx=0x7f21dc020580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:09.249 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20344 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:09.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc0425b0 2021-03-09T09:42:09.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0425b0 2021-03-09T09:42:09.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc09a070 con 0x7f21cc0425b0 2021-03-09T09:42:09.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.245+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:09.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.249+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:09.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:09.249+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:14.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:42:14.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:14.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6062 cs=0 l=1 rev1=1 rx=0x7f21dc0ac380 tx=0x7f21dc020580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:14.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6062 cs=0 l=1 rev1=1 rx=0x7f21dc0ac380 tx=0x7f21dc020580).stop 2021-03-09T09:42:14.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:14.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:14.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:14.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:14.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:14.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:14.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21c4003cb0 2021-03-09T09:42:14.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21cc042f20 2021-03-09T09:42:14.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:14.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:14.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:14.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:14.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:14.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:14.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:14.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:14.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:14.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:42:14.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.241+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03d0c0 con 0x7f21cc042f20 2021-03-09T09:42:14.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.245+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6072 cs=0 l=1 rev1=1 rx=0x7f21cc024490 tx=0x7f21dc0b6350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:14.253 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:14.245+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:14.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc042f20 2021-03-09T09:42:14.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028880 con 0x7f21cc042f20 2021-03-09T09:42:14.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc09a070 con 0x7f21cc042f20 2021-03-09T09:42:14.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.249+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:14.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.253+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:14.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:14.253+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:19.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:19.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:19.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6072 cs=0 l=1 rev1=1 rx=0x7f21cc024490 tx=0x7f21dc0b6350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:19.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6072 cs=0 l=1 rev1=1 rx=0x7f21cc024490 tx=0x7f21dc0b6350).stop 2021-03-09T09:42:19.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:19.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:19.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:19.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:19.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:19.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:19.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21c4003cb0 2021-03-09T09:42:19.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21cc0425b0 2021-03-09T09:42:19.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:19.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:19.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:19.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:19.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.245+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:19.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:19.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:19.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:19.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:19.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:42:19.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc0425b0 2021-03-09T09:42:19.263 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20418 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:19.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6083 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0b6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:19.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3d90 con 0x7f21cc0425b0 2021-03-09T09:42:19.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc0425b0 2021-03-09T09:42:19.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.253+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018d50 con 0x7f21cc0425b0 2021-03-09T09:42:19.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.253+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:19.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.257+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:19.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:19.257+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:21.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:21.718 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:24.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:24.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:24.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6083 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0b6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:24.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6083 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0b6200).stop 2021-03-09T09:42:24.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:24.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:24.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:24.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:24.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:24.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:24.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0578b0 con 0x7f21c4003cb0 2021-03-09T09:42:24.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21cc042f20 2021-03-09T09:42:24.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:24.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:24.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:24.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:24.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.249+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:24.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:24.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:24.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:24.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:24.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc042f20 2021-03-09T09:42:24.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:42:24.260 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20454 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:24.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.253+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6095 cs=0 l=1 rev1=1 rx=0x7f21dc0b4050 tx=0x7f21dc062bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:24.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083da0 con 0x7f21cc042f20 2021-03-09T09:42:24.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc021790 con 0x7f21cc042f20 2021-03-09T09:42:24.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d90 con 0x7f21cc042f20 2021-03-09T09:42:24.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.261+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:24.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.261+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:24.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:24.261+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:29.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:29.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:29.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6095 cs=0 l=1 rev1=1 rx=0x7f21dc0b4050 tx=0x7f21dc062bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:29.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6095 cs=0 l=1 rev1=1 rx=0x7f21dc0b4050 tx=0x7f21dc062bf0).stop 2021-03-09T09:42:29.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:29.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:29.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:29.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:29.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:29.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:29.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21c4003cb0 2021-03-09T09:42:29.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21cc0425b0 2021-03-09T09:42:29.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:29.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:29.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:29.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:29.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:29.263 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20492 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:29.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:29.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:29.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:29.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:29.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:42:29.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc0425b0 2021-03-09T09:42:29.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.257+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6107 cs=0 l=1 rev1=1 rx=0x7f21dc06aa90 tx=0x7f21dc056190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:29.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc021790 con 0x7f21cc0425b0 2021-03-09T09:42:29.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc021790 con 0x7f21cc0425b0 2021-03-09T09:42:29.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc0425b0 2021-03-09T09:42:29.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.265+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:29.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.265+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:29.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:29.265+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:34.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:34.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:34.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6107 cs=0 l=1 rev1=1 rx=0x7f21dc06aa90 tx=0x7f21dc056190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:34.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6107 cs=0 l=1 rev1=1 rx=0x7f21dc06aa90 tx=0x7f21dc056190).stop 2021-03-09T09:42:34.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:34.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:34.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:34.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:34.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:34.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:34.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21c4003cb0 2021-03-09T09:42:34.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21cc042f20 2021-03-09T09:42:34.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:34.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:34.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:34.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:34.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:34.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:34.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:34.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:34.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:34.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc042f20 2021-03-09T09:42:34.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0304a0 con 0x7f21cc042f20 2021-03-09T09:42:34.269 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20528 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:34.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.261+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6116 cs=0 l=1 rev1=1 rx=0x7f21cc054450 tx=0x7f21e4027e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:34.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.265+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009430 con 0x7f21cc042f20 2021-03-09T09:42:34.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.269+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f380 con 0x7f21cc042f20 2021-03-09T09:42:34.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.273+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21cc042f20 2021-03-09T09:42:34.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.273+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:34.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.273+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:34.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:34.273+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:39.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:39.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:39.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6116 cs=0 l=1 rev1=1 rx=0x7f21cc054450 tx=0x7f21e4027e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:39.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6116 cs=0 l=1 rev1=1 rx=0x7f21cc054450 tx=0x7f21e4027e00).stop 2021-03-09T09:42:39.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:39.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:39.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:39.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:39.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:39.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc005e30 2021-03-09T09:42:39.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21c4003cb0 2021-03-09T09:42:39.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b920 con 0x7f21cc0425b0 2021-03-09T09:42:39.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:39.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:39.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:39.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:39.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:39.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:39.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:39.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:42:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc0425b0 2021-03-09T09:42:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6127 cs=0 l=1 rev1=1 rx=0x7f21cc0592b0 tx=0x7f21dc069730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:39.273 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:39.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0562c0 con 0x7f21cc0425b0 2021-03-09T09:42:39.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06aac0 con 0x7f21cc0425b0 2021-03-09T09:42:39.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc0425b0 2021-03-09T09:42:39.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.268+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:39.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.272+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:39.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:39.272+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:44.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:44.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:44.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6127 cs=0 l=1 rev1=1 rx=0x7f21cc0592b0 tx=0x7f21dc069730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:44.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6127 cs=0 l=1 rev1=1 rx=0x7f21cc0592b0 tx=0x7f21dc069730).stop 2021-03-09T09:42:44.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:44.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:44.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:44.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:44.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:44.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6f0 con 0x7f21cc005e30 2021-03-09T09:42:44.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f230 con 0x7f21c4003cb0 2021-03-09T09:42:44.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21cc042f20 2021-03-09T09:42:44.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:44.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:44.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:44.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:44.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:44.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:44.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:44.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:44.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:44.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc042f20 2021-03-09T09:42:44.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:42:44.277 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20602 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:44.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6136 cs=0 l=1 rev1=1 rx=0x7f21dc0564e0 tx=0x7f21dc0509b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:44.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc042f20 2021-03-09T09:42:44.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06bbc0 con 0x7f21cc042f20 2021-03-09T09:42:44.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02bde0 con 0x7f21cc042f20 2021-03-09T09:42:44.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.272+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:44.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.276+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:44.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:44.276+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:49.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:49.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:49.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6136 cs=0 l=1 rev1=1 rx=0x7f21dc0564e0 tx=0x7f21dc0509b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:49.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6136 cs=0 l=1 rev1=1 rx=0x7f21dc0564e0 tx=0x7f21dc0509b0).stop 2021-03-09T09:42:49.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:49.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:49.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:49.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:49.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:49.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f230 con 0x7f21cc005e30 2021-03-09T09:42:49.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007980 con 0x7f21c4003cb0 2021-03-09T09:42:49.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0862a0 con 0x7f21cc0425b0 2021-03-09T09:42:49.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:49.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:49.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:49.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:49.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:49.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:49.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:49.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:49.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:49.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f000 con 0x7f21cc0425b0 2021-03-09T09:42:49.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0304a0 con 0x7f21cc0425b0 2021-03-09T09:42:49.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6147 cs=0 l=1 rev1=1 rx=0x7f21cc015520 tx=0x7f21e4028ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:49.281 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20640 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:49.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc0425b0 2021-03-09T09:42:49.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0425b0 2021-03-09T09:42:49.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40092e0 con 0x7f21cc0425b0 2021-03-09T09:42:49.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.276+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:49.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.280+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:49.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:49.280+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:51.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:51.767 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:42:54.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:54.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:54.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6147 cs=0 l=1 rev1=1 rx=0x7f21cc015520 tx=0x7f21e4028ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:54.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6147 cs=0 l=1 rev1=1 rx=0x7f21cc015520 tx=0x7f21e4028ec0).stop 2021-03-09T09:42:54.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:54.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:54.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:54.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:54.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:54.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f230 con 0x7f21cc005e30 2021-03-09T09:42:54.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0862a0 con 0x7f21c4003cb0 2021-03-09T09:42:54.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21cc042f20 2021-03-09T09:42:54.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:54.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:54.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:54.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:54.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:54.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:54.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:54.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:54.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:54.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:42:54.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:42:54.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6158 cs=0 l=1 rev1=1 rx=0x7f21cc04aba0 tx=0x7f21dc050c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:54.285 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:54.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc042f20 2021-03-09T09:42:54.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:42:54.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:42:54.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.280+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:54.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.280+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:54.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:54.284+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:42:59.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:59.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:59.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6158 cs=0 l=1 rev1=1 rx=0x7f21cc04aba0 tx=0x7f21dc050c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:59.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6158 cs=0 l=1 rev1=1 rx=0x7f21cc04aba0 tx=0x7f21dc050c50).stop 2021-03-09T09:42:59.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:59.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:59.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:59.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:59.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:59.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0864a0 con 0x7f21cc005e30 2021-03-09T09:42:59.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0145f0 con 0x7f21c4003cb0 2021-03-09T09:42:59.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0862a0 con 0x7f21cc0425b0 2021-03-09T09:42:59.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:42:59.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:59.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:59.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:42:59.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:59.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:59.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:59.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:59.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:59.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0425b0 2021-03-09T09:42:59.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc0425b0 2021-03-09T09:42:59.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6168 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e4016ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:59.289 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20714 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:42:59.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0425b0 2021-03-09T09:42:59.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0425b0 2021-03-09T09:42:59.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0425b0 2021-03-09T09:42:59.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.284+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:42:59.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:42:59.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:42:59.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:04.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:04.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:04.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6168 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e4016ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:04.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6168 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e4016ab0).stop 2021-03-09T09:43:04.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:04.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:04.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:04.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:04.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:04.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0524f0 con 0x7f21cc005e30 2021-03-09T09:43:04.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0145f0 con 0x7f21c4003cb0 2021-03-09T09:43:04.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:43:04.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:04.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:04.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:04.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:04.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:04.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:04.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:04.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:04.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:04.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057880 con 0x7f21cc042f20 2021-03-09T09:43:04.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:43:04.292 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20750 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:04.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6179 cs=0 l=1 rev1=1 rx=0x7f21dc00e040 tx=0x7f21dc0bb5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:04.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc042f20 2021-03-09T09:43:04.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050a20 con 0x7f21cc042f20 2021-03-09T09:43:04.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:43:04.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.288+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:04.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:04.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:04.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:09.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:09.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:09.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6179 cs=0 l=1 rev1=1 rx=0x7f21dc00e040 tx=0x7f21dc0bb5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:09.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6179 cs=0 l=1 rev1=1 rx=0x7f21dc00e040 tx=0x7f21dc0bb5a0).stop 2021-03-09T09:43:09.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:09.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:09.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:09.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:09.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:09.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b050 con 0x7f21cc005e30 2021-03-09T09:43:09.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21c4003cb0 2021-03-09T09:43:09.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:09.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052350 con 0x7f21cc0425b0 2021-03-09T09:43:09.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:09.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:09.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:09.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:09.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:09.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:09.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:09.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:09.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0524c0 con 0x7f21cc0425b0 2021-03-09T09:43:09.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc0425b0 2021-03-09T09:43:09.296 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20788 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:09.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6190 cs=0 l=1 rev1=1 rx=0x7f21cc035380 tx=0x7f21e40a6230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:09.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc0425b0 2021-03-09T09:43:09.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc0425b0 2021-03-09T09:43:09.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc0425b0 2021-03-09T09:43:09.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.296+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:09.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.296+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:09.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:09.296+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:14.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:14.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:14.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6190 cs=0 l=1 rev1=1 rx=0x7f21cc035380 tx=0x7f21e40a6230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:14.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 secure :-1 s=READY pgs=6190 cs=0 l=1 rev1=1 rx=0x7f21cc035380 tx=0x7f21e40a6230).stop 2021-03-09T09:43:14.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:14.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02ced0 unknown :-1 s=CLOSED pgs=6190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:14.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:14.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:14.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:14.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052350 con 0x7f21cc005e30 2021-03-09T09:43:14.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b050 con 0x7f21c4003cb0 2021-03-09T09:43:14.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f000 con 0x7f21cc042f20 2021-03-09T09:43:14.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:14.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:14.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:14.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:14.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:14.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:14.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:14.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:14.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:14.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:43:14.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0524c0 con 0x7f21cc042f20 2021-03-09T09:43:14.300 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20824 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:14.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.292+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f21e40a6380 tx=0x7f21e40a64c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:14.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21cc042f20 2021-03-09T09:43:14.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404aa20 con 0x7f21cc042f20 2021-03-09T09:43:14.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21cc042f20 2021-03-09T09:43:14.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.300+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:14.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:14.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:14.300+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:19.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:19.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:19.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f21e40a6380 tx=0x7f21e40a64c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:19.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f21e40a6380 tx=0x7f21e40a64c0).stop 2021-03-09T09:43:19.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:19.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:19.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:19.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:19.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:19.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21cc005e30 2021-03-09T09:43:19.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0542d0 con 0x7f21c4003cb0 2021-03-09T09:43:19.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052350 con 0x7f21cc0425b0 2021-03-09T09:43:19.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:19.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:19.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:19.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:19.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:19.302 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20862 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:19.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:19.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:19.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:19.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:19.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b050 con 0x7f21cc0425b0 2021-03-09T09:43:19.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057880 con 0x7f21cc0425b0 2021-03-09T09:43:19.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d5a0 secure :-1 s=READY pgs=6212 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:19.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0425b0 2021-03-09T09:43:19.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc029350 con 0x7f21cc0425b0 2021-03-09T09:43:19.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc099d70 con 0x7f21cc0425b0 2021-03-09T09:43:19.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.304+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:19.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.304+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:19.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:19.304+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:21.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:21.810 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:24.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:24.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:24.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d5a0 secure :-1 s=READY pgs=6212 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:24.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d5a0 secure :-1 s=READY pgs=6212 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).stop 2021-03-09T09:43:24.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc02d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:24.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc02d5a0 unknown :-1 s=CLOSED pgs=6212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:24.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:24.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:24.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:24.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21cc005e30 2021-03-09T09:43:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045f10 con 0x7f21c4003cb0 2021-03-09T09:43:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0555e0 con 0x7f21cc042f20 2021-03-09T09:43:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:24.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:24.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:24.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:24.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:24.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:24.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:24.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:24.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f860 con 0x7f21cc042f20 2021-03-09T09:43:24.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b050 con 0x7f21cc042f20 2021-03-09T09:43:24.307 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20898 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:24.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.300+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6222 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc007a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:24.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc029350 con 0x7f21cc042f20 2021-03-09T09:43:24.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc029350 con 0x7f21cc042f20 2021-03-09T09:43:24.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.308+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21cc042f20 2021-03-09T09:43:24.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.308+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:24.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:24.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:24.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:29.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:29.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:29.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6222 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc007a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:29.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6222 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc007a10).stop 2021-03-09T09:43:29.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:29.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:29.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21cc005e30 2021-03-09T09:43:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0555e0 con 0x7f21c4003cb0 2021-03-09T09:43:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:43:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:29.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:29.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f000 con 0x7f21cc0425b0 2021-03-09T09:43:29.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f860 con 0x7f21cc0425b0 2021-03-09T09:43:29.311 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20936 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc045bd0 secure :-1 s=READY pgs=6233 cs=0 l=1 rev1=1 rx=0x7f21dc007bd0 tx=0x7f21dc056200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bb440 con 0x7f21cc0425b0 2021-03-09T09:43:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0425b0 2021-03-09T09:43:29.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21cc0425b0 2021-03-09T09:43:29.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.311+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:29.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.311+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:29.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:29.311+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:34.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:34.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:34.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc045bd0 secure :-1 s=READY pgs=6233 cs=0 l=1 rev1=1 rx=0x7f21dc007bd0 tx=0x7f21dc056200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:34.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc045bd0 secure :-1 s=READY pgs=6233 cs=0 l=1 rev1=1 rx=0x7f21dc007bd0 tx=0x7f21dc056200).stop 2021-03-09T09:43:34.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:34.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=6233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:34.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:34.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:34.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:34.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026c10 con 0x7f21cc005e30 2021-03-09T09:43:34.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21c4003cb0 2021-03-09T09:43:34.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0555e0 con 0x7f21cc042f20 2021-03-09T09:43:34.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:34.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:34.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:34.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:34.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:34.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:34.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:34.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc022d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:34.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc022d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:43:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc042f20 2021-03-09T09:43:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6243 cs=0 l=1 rev1=1 rx=0x7f21dc056510 tx=0x7f21dc020510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:34.316 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20972 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:34.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007e70 con 0x7f21cc042f20 2021-03-09T09:43:34.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:43:34.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:43:34.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.311+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:34.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.311+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:34.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:34.311+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:39.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:39.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:39.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6243 cs=0 l=1 rev1=1 rx=0x7f21dc056510 tx=0x7f21dc020510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:39.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6243 cs=0 l=1 rev1=1 rx=0x7f21dc056510 tx=0x7f21dc020510).stop 2021-03-09T09:43:39.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:39.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:39.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:39.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc022c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:39.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:39.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02be20 con 0x7f21cc005e30 2021-03-09T09:43:39.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026c10 con 0x7f21c4003cb0 2021-03-09T09:43:39.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:39.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21cc0425b0 2021-03-09T09:43:39.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:39.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:39.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc022c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:39.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:39.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:39.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:39.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:39.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:39.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0555e0 con 0x7f21cc0425b0 2021-03-09T09:43:39.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0524c0 con 0x7f21cc0425b0 2021-03-09T09:43:39.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc022c40 secure :-1 s=READY pgs=6254 cs=0 l=1 rev1=1 rx=0x7f21e4029040 tx=0x7f21e4047170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:39.321 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:39.315+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21010 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:39.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21cc0425b0 2021-03-09T09:43:39.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404ac30 con 0x7f21cc0425b0 2021-03-09T09:43:39.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404ac30 con 0x7f21cc0425b0 2021-03-09T09:43:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.323+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:39.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:44.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc022c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:44.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc022c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:44.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc022c40 secure :-1 s=READY pgs=6254 cs=0 l=1 rev1=1 rx=0x7f21e4029040 tx=0x7f21e4047170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:44.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc022c40 secure :-1 s=READY pgs=6254 cs=0 l=1 rev1=1 rx=0x7f21e4029040 tx=0x7f21e4047170).stop 2021-03-09T09:43:44.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc022c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:44.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc022c40 unknown :-1 s=CLOSED pgs=6254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:44.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:44.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:44.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:44.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026c10 con 0x7f21cc005e30 2021-03-09T09:43:44.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21c4003cb0 2021-03-09T09:43:44.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045320 con 0x7f21cc042f20 2021-03-09T09:43:44.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:44.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:44.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:44.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:44.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:44.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:44.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:44.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:44.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:44.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02bf70 con 0x7f21cc042f20 2021-03-09T09:43:44.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0555e0 con 0x7f21cc042f20 2021-03-09T09:43:44.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6264 cs=0 l=1 rev1=1 rx=0x7f21e404aff0 tx=0x7f21e4007390).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:44.324 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:44.319+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21046 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:44.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f4e0 con 0x7f21cc042f20 2021-03-09T09:43:44.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21cc042f20 2021-03-09T09:43:44.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21cc042f20 2021-03-09T09:43:44.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.323+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:44.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:44.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:44.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:49.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:49.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:49.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6264 cs=0 l=1 rev1=1 rx=0x7f21e404aff0 tx=0x7f21e4007390).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:49.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6264 cs=0 l=1 rev1=1 rx=0x7f21e404aff0 tx=0x7f21e4007390).stop 2021-03-09T09:43:49.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:49.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:49.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:49.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc0524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:49.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:49.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026c10 con 0x7f21cc005e30 2021-03-09T09:43:49.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f2a0 con 0x7f21c4003cb0 2021-03-09T09:43:49.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f30 con 0x7f21cc0425b0 2021-03-09T09:43:49.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:49.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:49.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:49.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:49.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc0524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:49.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:49.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:49.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:49.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:49.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0555e0 con 0x7f21cc0425b0 2021-03-09T09:43:49.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02bf70 con 0x7f21cc0425b0 2021-03-09T09:43:49.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc0524c0 secure :-1 s=READY pgs=6276 cs=0 l=1 rev1=1 rx=0x7f21e40470d0 tx=0x7f21e4007160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:49.330 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:49.323+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21084 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:49.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40bb080 con 0x7f21cc0425b0 2021-03-09T09:43:49.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401fc40 con 0x7f21cc0425b0 2021-03-09T09:43:49.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401fdb0 con 0x7f21cc0425b0 2021-03-09T09:43:49.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.331+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:49.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:49.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:49.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:51.872 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:51.877 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:43:54.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc0524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:43:54.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc0524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:54.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc0524c0 secure :-1 s=READY pgs=6276 cs=0 l=1 rev1=1 rx=0x7f21e40470d0 tx=0x7f21e4007160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:54.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc0524c0 secure :-1 s=READY pgs=6276 cs=0 l=1 rev1=1 rx=0x7f21e40470d0 tx=0x7f21e4007160).stop 2021-03-09T09:43:54.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc0524c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:54.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc0524c0 unknown :-1 s=CLOSED pgs=6276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:54.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:54.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:54.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:54.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045300 con 0x7f21cc005e30 2021-03-09T09:43:54.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T09:43:54.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0577e0 con 0x7f21cc042f20 2021-03-09T09:43:54.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:54.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:54.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:54.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:54.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:54.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:54.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:54.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:54.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:54.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057950 con 0x7f21cc042f20 2021-03-09T09:43:54.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0555e0 con 0x7f21cc042f20 2021-03-09T09:43:54.338 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21119 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:54.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6286 cs=0 l=1 rev1=1 rx=0x7f21cc04af40 tx=0x7f21e400f480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:54.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a4070 con 0x7f21cc042f20 2021-03-09T09:43:54.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T09:43:54.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401fc40 con 0x7f21cc042f20 2021-03-09T09:43:54.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.335+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:54.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:54.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:54.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:43:59.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:59.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:59.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6286 cs=0 l=1 rev1=1 rx=0x7f21cc04af40 tx=0x7f21e400f480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:59.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 secure :-1 s=READY pgs=6286 cs=0 l=1 rev1=1 rx=0x7f21cc04af40 tx=0x7f21e400f480).stop 2021-03-09T09:43:59.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc042990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:59.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc042990 unknown :-1 s=CLOSED pgs=6286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:59.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:59.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:59.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc052490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:59.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0577e0 con 0x7f21cc005e30 2021-03-09T09:43:59.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045300 con 0x7f21c4003cb0 2021-03-09T09:43:59.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0555e0 con 0x7f21cc0425b0 2021-03-09T09:43:59.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:43:59.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:59.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:43:59.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:59.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc052490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:59.334 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:43:59.327+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21158 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:43:59.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:59.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:59.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:59.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:59.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc0425b0 2021-03-09T09:43:59.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc0425b0 2021-03-09T09:43:59.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc052490 secure :-1 s=READY pgs=6298 cs=0 l=1 rev1=1 rx=0x7f21cc01ac20 tx=0x7f21dc07cc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:59.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21cc0425b0 2021-03-09T09:43:59.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0425b0 2021-03-09T09:43:59.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc0425b0 2021-03-09T09:43:59.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.335+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:43:59.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.339+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:43:59.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:43:59.339+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:04.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc052490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:04.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc052490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:04.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc052490 secure :-1 s=READY pgs=6298 cs=0 l=1 rev1=1 rx=0x7f21cc01ac20 tx=0x7f21dc07cc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:04.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc052490 secure :-1 s=READY pgs=6298 cs=0 l=1 rev1=1 rx=0x7f21cc01ac20 tx=0x7f21dc07cc90).stop 2021-03-09T09:44:04.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 msgr2=0x7f21cc052490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:04.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0425b0 0x7f21cc052490 unknown :-1 s=CLOSED pgs=6298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:04.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:04.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:04.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:04.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0216c0 con 0x7f21cc005e30 2021-03-09T09:44:04.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0577e0 con 0x7f21c4003cb0 2021-03-09T09:44:04.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045300 con 0x7f21cc042f20 2021-03-09T09:44:04.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:04.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:04.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:04.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:04.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:04.339 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:04.331+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21193 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:04.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:04.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:04.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:04.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:04.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0555e0 con 0x7f21cc042f20 2021-03-09T09:44:04.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc042f20 2021-03-09T09:44:04.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d5a0 secure :-1 s=READY pgs=6312 cs=0 l=1 rev1=1 rx=0x7f21cc052980 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:04.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085210 con 0x7f21cc042f20 2021-03-09T09:44:04.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:44:04.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:44:04.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.343+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:04.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.343+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:04.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:04.343+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:09.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:09.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:09.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d5a0 secure :-1 s=READY pgs=6312 cs=0 l=1 rev1=1 rx=0x7f21cc052980 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:09.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d5a0 secure :-1 s=READY pgs=6312 cs=0 l=1 rev1=1 rx=0x7f21cc052980 tx=0x7f21dc08a120).stop 2021-03-09T09:44:09.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:09.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02d5a0 unknown :-1 s=CLOSED pgs=6312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:09.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:09.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:09.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:09.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:09.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0577e0 con 0x7f21cc005e30 2021-03-09T09:44:09.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:09.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045300 con 0x7f21c4003cb0 2021-03-09T09:44:09.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f000 con 0x7f21cc052490 2021-03-09T09:44:09.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:09.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:09.341 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21232 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:09.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.335+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:09.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:09.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:09.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:09.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:09.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:44:09.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057950 con 0x7f21cc052490 2021-03-09T09:44:09.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 secure :-1 s=READY pgs=6323 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e401a2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:09.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc052490 2021-03-09T09:44:09.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc052490 2021-03-09T09:44:09.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:44:09.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.347+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:09.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.347+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:09.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:09.347+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:14.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:14.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:14.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 secure :-1 s=READY pgs=6323 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e401a2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:14.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 secure :-1 s=READY pgs=6323 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e401a2e0).stop 2021-03-09T09:44:14.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0425b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:14.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 unknown :-1 s=CLOSED pgs=6323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:14.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:14.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:14.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:14.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049a40 con 0x7f21cc005e30 2021-03-09T09:44:14.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0354a0 con 0x7f21c4003cb0 2021-03-09T09:44:14.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f80 con 0x7f21cc042f20 2021-03-09T09:44:14.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:14.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:14.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:14.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:14.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:14.348 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:14.339+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21267 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:14.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:14.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:14.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:14.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:14.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f000 con 0x7f21cc042f20 2021-03-09T09:44:14.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0555e0 con 0x7f21cc042f20 2021-03-09T09:44:14.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.343+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 secure :-1 s=READY pgs=6335 cs=0 l=1 rev1=1 rx=0x7f21cc01df70 tx=0x7f21dc083ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:14.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:44:14.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:44:14.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:44:14.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.351+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:14.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.351+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:14.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:14.351+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:19.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:19.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:19.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 secure :-1 s=READY pgs=6335 cs=0 l=1 rev1=1 rx=0x7f21cc01df70 tx=0x7f21dc083ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:19.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 secure :-1 s=READY pgs=6335 cs=0 l=1 rev1=1 rx=0x7f21cc01df70 tx=0x7f21dc083ef0).stop 2021-03-09T09:44:19.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:19.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 unknown :-1 s=CLOSED pgs=6335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:19.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:19.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:19.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:19.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0354a0 con 0x7f21cc005e30 2021-03-09T09:44:19.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057980 con 0x7f21c4003cb0 2021-03-09T09:44:19.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049be0 con 0x7f21cc052490 2021-03-09T09:44:19.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:19.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:19.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:19.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:19.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:19.349 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:19.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:19.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:19.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:19.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:19.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091440 con 0x7f21cc052490 2021-03-09T09:44:19.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:44:19.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 secure :-1 s=READY pgs=6341 cs=0 l=1 rev1=1 rx=0x7f21e40470a0 tx=0x7f21e4047cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:19.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003af0 con 0x7f21cc052490 2021-03-09T09:44:19.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4003af0 con 0x7f21cc052490 2021-03-09T09:44:19.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4047d80 con 0x7f21cc052490 2021-03-09T09:44:19.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.354+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:19.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:19.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:19.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:21.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:21.936 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:24.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:24.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:24.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 secure :-1 s=READY pgs=6341 cs=0 l=1 rev1=1 rx=0x7f21e40470a0 tx=0x7f21e4047cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:24.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 secure :-1 s=READY pgs=6341 cs=0 l=1 rev1=1 rx=0x7f21e40470a0 tx=0x7f21e4047cb0).stop 2021-03-09T09:44:24.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0425b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:24.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0425b0 unknown :-1 s=CLOSED pgs=6341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:24.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:24.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:24.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:24.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0354a0 con 0x7f21cc005e30 2021-03-09T09:44:24.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049be0 con 0x7f21c4003cb0 2021-03-09T09:44:24.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057980 con 0x7f21cc042f20 2021-03-09T09:44:24.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:24.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:24.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:24.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:24.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:24.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:24.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:24.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:24.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:24.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d570 con 0x7f21cc042f20 2021-03-09T09:44:24.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc042f20 2021-03-09T09:44:24.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 secure :-1 s=READY pgs=6350 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc087520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:24.355 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:24.346+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21341 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:24.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00aeb0 con 0x7f21cc042f20 2021-03-09T09:44:24.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:44:24.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00aeb0 con 0x7f21cc042f20 2021-03-09T09:44:24.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.354+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:24.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:24.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:24.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:29.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:29.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:29.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 secure :-1 s=READY pgs=6350 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc087520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:29.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 secure :-1 s=READY pgs=6350 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc087520).stop 2021-03-09T09:44:29.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:29.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013230 unknown :-1 s=CLOSED pgs=6350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:29.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:29.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:29.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:29.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0574d0 con 0x7f21cc005e30 2021-03-09T09:44:29.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0962a0 con 0x7f21c4003cb0 2021-03-09T09:44:29.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:29.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095570 con 0x7f21cc052490 2021-03-09T09:44:29.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:29.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:29.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:29.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:29.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:29.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:29.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:29.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:29.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f000 con 0x7f21cc052490 2021-03-09T09:44:29.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091440 con 0x7f21cc052490 2021-03-09T09:44:29.359 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:29.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6360 cs=0 l=1 rev1=1 rx=0x7f21e408f3f0 tx=0x7f21e4005f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:29.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc052490 2021-03-09T09:44:29.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc052490 2021-03-09T09:44:29.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc052490 2021-03-09T09:44:29.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.358+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:29.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:29.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:29.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:34.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:34.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:34.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6360 cs=0 l=1 rev1=1 rx=0x7f21e408f3f0 tx=0x7f21e4005f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:34.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6360 cs=0 l=1 rev1=1 rx=0x7f21e408f3f0 tx=0x7f21e4005f20).stop 2021-03-09T09:44:34.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:34.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=CLOSED pgs=6360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:34.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc052f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:34.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:34.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:34.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041020 con 0x7f21cc005e30 2021-03-09T09:44:34.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0354a0 con 0x7f21c4003cb0 2021-03-09T09:44:34.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21cc042f20 2021-03-09T09:44:34.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:34.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:34.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:34.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:34.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc052f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:34.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:34.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:34.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:34.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:34.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023390 con 0x7f21cc042f20 2021-03-09T09:44:34.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d570 con 0x7f21cc042f20 2021-03-09T09:44:34.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc052f80 secure :-1 s=READY pgs=6371 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:34.363 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21415 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:34.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02fcc0 con 0x7f21cc042f20 2021-03-09T09:44:34.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc042f20 2021-03-09T09:44:34.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:44:34.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.362+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:34.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:34.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:34.362+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:39.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc052f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:39.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc052f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:39.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc052f80 secure :-1 s=READY pgs=6371 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:39.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc052f80 secure :-1 s=READY pgs=6371 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc077de0).stop 2021-03-09T09:44:39.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc052f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:39.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc052f80 unknown :-1 s=CLOSED pgs=6371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:39.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:39.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:39.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:39.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21cc005e30 2021-03-09T09:44:39.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0354a0 con 0x7f21c4003cb0 2021-03-09T09:44:39.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21cc052490 2021-03-09T09:44:39.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:39.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:39.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:39.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:39.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:39.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:39.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:39.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:39.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:39.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095710 con 0x7f21cc052490 2021-03-09T09:44:39.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023390 con 0x7f21cc052490 2021-03-09T09:44:39.367 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:39.358+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21454 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:39.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6383 cs=0 l=1 rev1=1 rx=0x7f21dc008010 tx=0x7f21dc0b3120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:39.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0baeb0 con 0x7f21cc052490 2021-03-09T09:44:39.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:44:39.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0baca0 con 0x7f21cc052490 2021-03-09T09:44:39.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.366+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:39.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:39.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:39.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:44.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:44.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:44.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6383 cs=0 l=1 rev1=1 rx=0x7f21dc008010 tx=0x7f21dc0b3120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:44.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6383 cs=0 l=1 rev1=1 rx=0x7f21dc008010 tx=0x7f21dc0b3120).stop 2021-03-09T09:44:44.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:44.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=CLOSED pgs=6383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:44.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:44.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:44.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:44.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21cc005e30 2021-03-09T09:44:44.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:44.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:44.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21c4003cb0 2021-03-09T09:44:44.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091430 con 0x7f21cc042f20 2021-03-09T09:44:44.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:44.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:44.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:44.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:44.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:44.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:44.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:44.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023390 con 0x7f21cc042f20 2021-03-09T09:44:44.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095710 con 0x7f21cc042f20 2021-03-09T09:44:44.371 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21489 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:44.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d2c0 secure :-1 s=READY pgs=6392 cs=0 l=1 rev1=1 rx=0x7f21dc0b32e0 tx=0x7f21dc062120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:44.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21cc042f20 2021-03-09T09:44:44.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21cc042f20 2021-03-09T09:44:44.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0622f0 con 0x7f21cc042f20 2021-03-09T09:44:44.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.374+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:44.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.374+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:44.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:44.374+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:49.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:49.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:49.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d2c0 secure :-1 s=READY pgs=6392 cs=0 l=1 rev1=1 rx=0x7f21dc0b32e0 tx=0x7f21dc062120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:49.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d2c0 secure :-1 s=READY pgs=6392 cs=0 l=1 rev1=1 rx=0x7f21dc0b32e0 tx=0x7f21dc062120).stop 2021-03-09T09:44:49.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:49.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d2c0 unknown :-1 s=CLOSED pgs=6392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:49.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:49.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:49.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:49.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21cc005e30 2021-03-09T09:44:49.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21c4003cb0 2021-03-09T09:44:49.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b1f0 con 0x7f21cc052490 2021-03-09T09:44:49.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:49.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:49.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:49.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:49.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:49.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:49.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:49.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:49.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:49.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:44:49.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023390 con 0x7f21cc052490 2021-03-09T09:44:49.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6404 cs=0 l=1 rev1=1 rx=0x7f21dc029630 tx=0x7f21dc015a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:49.375 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:49.366+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21528 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:49.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc064950 con 0x7f21cc052490 2021-03-09T09:44:49.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:44:49.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02f6b0 con 0x7f21cc052490 2021-03-09T09:44:49.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.374+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:49.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.374+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:49.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:49.374+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:51.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:51.977 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:44:54.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:54.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:54.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6404 cs=0 l=1 rev1=1 rx=0x7f21dc029630 tx=0x7f21dc015a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:54.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 secure :-1 s=READY pgs=6404 cs=0 l=1 rev1=1 rx=0x7f21dc029630 tx=0x7f21dc015a20).stop 2021-03-09T09:44:54.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc042570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:54.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc042570 unknown :-1 s=CLOSED pgs=6404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:54.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:54.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:54.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:54.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cb60 con 0x7f21cc005e30 2021-03-09T09:44:54.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00e940 con 0x7f21c4003cb0 2021-03-09T09:44:54.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21cc042f20 2021-03-09T09:44:54.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:54.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:54.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:54.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:54.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:54.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:54.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:54.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:54.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:54.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023390 con 0x7f21cc042f20 2021-03-09T09:44:54.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0265f0 con 0x7f21cc042f20 2021-03-09T09:44:54.379 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:54.370+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21564 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:54.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d220 secure :-1 s=READY pgs=6415 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0649c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:54.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc042f20 2021-03-09T09:44:54.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21cc042f20 2021-03-09T09:44:54.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02f6b0 con 0x7f21cc042f20 2021-03-09T09:44:54.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.378+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:54.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.382+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:54.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:54.382+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:44:59.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:59.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d220 secure :-1 s=READY pgs=6415 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0649c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d220 secure :-1 s=READY pgs=6415 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0649c0).stop 2021-03-09T09:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d220 unknown :-1 s=CLOSED pgs=6415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:59.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:59.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:59.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:59.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21cc005e30 2021-03-09T09:44:59.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21c4003cb0 2021-03-09T09:44:59.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21cc052490 2021-03-09T09:44:59.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:44:59.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:59.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:44:59.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:59.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:59.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:59.382 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:44:59.374+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21599 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:44:59.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:59.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:59.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:59.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:44:59.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023390 con 0x7f21cc052490 2021-03-09T09:44:59.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e7a0 secure :-1 s=READY pgs=6427 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc056ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:59.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc052490 2021-03-09T09:44:59.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21cc052490 2021-03-09T09:44:59.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b8070 con 0x7f21cc052490 2021-03-09T09:44:59.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.390+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:44:59.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:44:59.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:44:59.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:04.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:04.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:04.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e7a0 secure :-1 s=READY pgs=6427 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc056ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:04.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e7a0 secure :-1 s=READY pgs=6427 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc056ad0).stop 2021-03-09T09:45:04.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e7a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:04.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e7a0 unknown :-1 s=CLOSED pgs=6427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:04.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:04.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:04.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:04.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027300 con 0x7f21cc005e30 2021-03-09T09:45:04.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21c4003cb0 2021-03-09T09:45:04.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21cc042f20 2021-03-09T09:45:04.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:04.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:04.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:04.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:04.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:04.386 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:04.378+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21638 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:04.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:04.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:04.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:04.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:04.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023390 con 0x7f21cc042f20 2021-03-09T09:45:04.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f000 con 0x7f21cc042f20 2021-03-09T09:45:04.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6437 cs=0 l=1 rev1=1 rx=0x7f21cc027b80 tx=0x7f21e4016b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:04.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T09:45:04.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T09:45:04.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408f160 con 0x7f21cc042f20 2021-03-09T09:45:04.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.390+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:04.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:04.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:04.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:09.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:09.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:09.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6437 cs=0 l=1 rev1=1 rx=0x7f21cc027b80 tx=0x7f21e4016b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:09.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6437 cs=0 l=1 rev1=1 rx=0x7f21cc027b80 tx=0x7f21e4016b20).stop 2021-03-09T09:45:09.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:09.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=CLOSED pgs=6437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:09.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:09.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:09.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:09.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21cc005e30 2021-03-09T09:45:09.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023600 con 0x7f21c4003cb0 2021-03-09T09:45:09.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096f40 con 0x7f21cc052490 2021-03-09T09:45:09.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:09.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:09.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:09.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:09.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:09.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:09.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:09.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:09.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:09.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ddc0 con 0x7f21cc052490 2021-03-09T09:45:09.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:45:09.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e460 secure :-1 s=READY pgs=6446 cs=0 l=1 rev1=1 rx=0x7f21cc094560 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:09.392 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:09.382+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21673 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:09.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3310 con 0x7f21cc052490 2021-03-09T09:45:09.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a90 con 0x7f21cc052490 2021-03-09T09:45:09.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b8070 con 0x7f21cc052490 2021-03-09T09:45:09.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.390+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:09.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:09.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:09.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:14.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:14.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:14.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e460 secure :-1 s=READY pgs=6446 cs=0 l=1 rev1=1 rx=0x7f21cc094560 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e460 secure :-1 s=READY pgs=6446 cs=0 l=1 rev1=1 rx=0x7f21cc094560 tx=0x7f21dc08a120).stop 2021-03-09T09:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e460 unknown :-1 s=CLOSED pgs=6446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc049850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053640 con 0x7f21cc005e30 2021-03-09T09:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b1f0 con 0x7f21c4003cb0 2021-03-09T09:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21cc042f20 2021-03-09T09:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:14.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc049850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:14.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023390 con 0x7f21cc042f20 2021-03-09T09:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc049850 secure :-1 s=READY pgs=6456 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e40a4f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:14.395 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:14.389+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21712 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:14.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409c050 con 0x7f21cc042f20 2021-03-09T09:45:14.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4047d20 con 0x7f21cc042f20 2021-03-09T09:45:14.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T09:45:14.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.393+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:14.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.393+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:14.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:14.393+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:19.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc049850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:19.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc049850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:19.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc049850 secure :-1 s=READY pgs=6456 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e40a4f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:19.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc049850 secure :-1 s=READY pgs=6456 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e40a4f00).stop 2021-03-09T09:45:19.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc049850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:19.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc049850 unknown :-1 s=CLOSED pgs=6456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:19.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:19.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:19.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:19.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21cc005e30 2021-03-09T09:45:19.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T09:45:19.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:45:19.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:19.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:19.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:19.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:19.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:19.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:19.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026730 con 0x7f21cc052490 2021-03-09T09:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ddc0 con 0x7f21cc052490 2021-03-09T09:45:19.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e120 secure :-1 s=READY pgs=6467 cs=0 l=1 rev1=1 rx=0x7f21cc02ce90 tx=0x7f21dc0ab1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:19.400 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:19.393+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21747 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:19.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21cc052490 2021-03-09T09:45:19.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc052490 2021-03-09T09:45:19.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ab4a0 con 0x7f21cc052490 2021-03-09T09:45:19.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.397+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:19.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:19.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:19.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:22.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:22.022 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:24.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:24.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:24.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e120 secure :-1 s=READY pgs=6467 cs=0 l=1 rev1=1 rx=0x7f21cc02ce90 tx=0x7f21dc0ab1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:24.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e120 secure :-1 s=READY pgs=6467 cs=0 l=1 rev1=1 rx=0x7f21cc02ce90 tx=0x7f21dc0ab1d0).stop 2021-03-09T09:45:24.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc00e120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:24.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc00e120 unknown :-1 s=CLOSED pgs=6467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:24.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:24.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:24.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:24.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc005e30 2021-03-09T09:45:24.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21c4003cb0 2021-03-09T09:45:24.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:45:24.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:24.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:24.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:24.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:24.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:24.403 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21786 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:24.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:24.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:24.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:24.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:24.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ddc0 con 0x7f21cc042f20 2021-03-09T09:45:24.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026730 con 0x7f21cc042f20 2021-03-09T09:45:24.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6482 cs=0 l=1 rev1=1 rx=0x7f21cc095450 tx=0x7f21dc085e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:24.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00d220 con 0x7f21cc042f20 2021-03-09T09:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00d380 con 0x7f21cc042f20 2021-03-09T09:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.405+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:24.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.409+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:24.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:24.409+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:29.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:29.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:29.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6482 cs=0 l=1 rev1=1 rx=0x7f21cc095450 tx=0x7f21dc085e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:29.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6482 cs=0 l=1 rev1=1 rx=0x7f21cc095450 tx=0x7f21dc085e70).stop 2021-03-09T09:45:29.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:29.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=CLOSED pgs=6482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:29.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc04fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:29.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:29.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:29.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21cc005e30 2021-03-09T09:45:29.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21c4003cb0 2021-03-09T09:45:29.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026730 con 0x7f21cc052490 2021-03-09T09:45:29.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:29.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:29.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:29.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:29.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc04fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:29.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:29.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:29.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:29.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:29.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc050260 con 0x7f21cc052490 2021-03-09T09:45:29.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ddc0 con 0x7f21cc052490 2021-03-09T09:45:29.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc04fd70 secure :-1 s=READY pgs=6489 cs=0 l=1 rev1=1 rx=0x7f21dc0621d0 tx=0x7f21dc0322d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:29.408 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:29.401+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21821 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:29.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc052490 2021-03-09T09:45:29.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc052490 2021-03-09T09:45:29.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc052490 2021-03-09T09:45:29.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.405+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:29.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.405+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:29.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:29.405+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:34.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc04fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:34.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc04fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:34.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc04fd70 secure :-1 s=READY pgs=6489 cs=0 l=1 rev1=1 rx=0x7f21dc0621d0 tx=0x7f21dc0322d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:34.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc04fd70 secure :-1 s=READY pgs=6489 cs=0 l=1 rev1=1 rx=0x7f21dc0621d0 tx=0x7f21dc0322d0).stop 2021-03-09T09:45:34.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc04fd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:34.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc04fd70 unknown :-1 s=CLOSED pgs=6489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:34.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:34.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:34.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:34.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c0e0 con 0x7f21cc005e30 2021-03-09T09:45:34.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21c4003cb0 2021-03-09T09:45:34.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:45:34.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:34.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:34.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:34.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:34.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:34.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:34.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:34.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:34.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:34.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ddc0 con 0x7f21cc042f20 2021-03-09T09:45:34.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc050260 con 0x7f21cc042f20 2021-03-09T09:45:34.412 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:34.405+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21860 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:34.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6499 cs=0 l=1 rev1=1 rx=0x7f21dc0330a0 tx=0x7f21dc018ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:34.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc042f20 2021-03-09T09:45:34.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc029300 con 0x7f21cc042f20 2021-03-09T09:45:34.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:45:34.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.409+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:34.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.409+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:34.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:34.409+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:39.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:39.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:39.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6499 cs=0 l=1 rev1=1 rx=0x7f21dc0330a0 tx=0x7f21dc018ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:39.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 secure :-1 s=READY pgs=6499 cs=0 l=1 rev1=1 rx=0x7f21dc0330a0 tx=0x7f21dc018ef0).stop 2021-03-09T09:45:39.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01d8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:39.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01d8d0 unknown :-1 s=CLOSED pgs=6499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:39.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:39.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:39.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:39.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:45:39.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9e0 con 0x7f21c4003cb0 2021-03-09T09:45:39.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:45:39.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:39.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:39.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:39.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:39.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:39.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:39.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:39.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:39.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:39.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026730 con 0x7f21cc052490 2021-03-09T09:45:39.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ddc0 con 0x7f21cc052490 2021-03-09T09:45:39.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6510 cs=0 l=1 rev1=1 rx=0x7f21cc0363d0 tx=0x7f21dc0b3120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:39.417 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:39.409+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21891 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:39.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02bf00 con 0x7f21cc052490 2021-03-09T09:45:39.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02bf00 con 0x7f21cc052490 2021-03-09T09:45:39.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:45:39.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.417+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:39.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.417+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:39.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:39.417+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:44.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:44.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:44.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6510 cs=0 l=1 rev1=1 rx=0x7f21cc0363d0 tx=0x7f21dc0b3120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:44.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6510 cs=0 l=1 rev1=1 rx=0x7f21cc0363d0 tx=0x7f21dc0b3120).stop 2021-03-09T09:45:44.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:44.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=CLOSED pgs=6510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:44.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:44.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:44.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:44.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc005e30 2021-03-09T09:45:44.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T09:45:44.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a2d0 con 0x7f21cc042f20 2021-03-09T09:45:44.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:44.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:44.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:44.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:44.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:44.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:44.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:44.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:44.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:44.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d8d0 con 0x7f21cc042f20 2021-03-09T09:45:44.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026730 con 0x7f21cc042f20 2021-03-09T09:45:44.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6520 cs=0 l=1 rev1=1 rx=0x7f21cc053500 tx=0x7f21dc085f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:44.421 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21934 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:44.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc042f20 2021-03-09T09:45:44.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc072e70 con 0x7f21cc042f20 2021-03-09T09:45:44.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:45:44.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.421+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:44.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:44.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:44.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:49.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:49.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:49.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6520 cs=0 l=1 rev1=1 rx=0x7f21cc053500 tx=0x7f21dc085f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:49.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6520 cs=0 l=1 rev1=1 rx=0x7f21cc053500 tx=0x7f21dc085f90).stop 2021-03-09T09:45:49.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:49.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=CLOSED pgs=6520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:49.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:49.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:49.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:49.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:45:49.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057c90 con 0x7f21c4003cb0 2021-03-09T09:45:49.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094510 con 0x7f21cc052490 2021-03-09T09:45:49.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:49.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:49.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:49.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:49.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:49.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:49.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:49.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:49.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:49.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08a2b0 con 0x7f21cc052490 2021-03-09T09:45:49.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d8d0 con 0x7f21cc052490 2021-03-09T09:45:49.424 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21968 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:49.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6531 cs=0 l=1 rev1=1 rx=0x7f21dc00f040 tx=0x7f21dc097ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:49.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc052490 2021-03-09T09:45:49.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031ff0 con 0x7f21cc052490 2021-03-09T09:45:49.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc052490 2021-03-09T09:45:49.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.425+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:49.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.425+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:49.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:49.425+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:52.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:52.063 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:45:54.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:54.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:54.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6531 cs=0 l=1 rev1=1 rx=0x7f21dc00f040 tx=0x7f21dc097ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:54.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6531 cs=0 l=1 rev1=1 rx=0x7f21dc00f040 tx=0x7f21dc097ea0).stop 2021-03-09T09:45:54.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:54.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=CLOSED pgs=6531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:54.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:54.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:54.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:54.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:45:54.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094510 con 0x7f21c4003cb0 2021-03-09T09:45:54.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053a00 con 0x7f21cc042f20 2021-03-09T09:45:54.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:54.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:54.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:54.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:54.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:54.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:54.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:54.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:54.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:54.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04f8e0 con 0x7f21cc042f20 2021-03-09T09:45:54.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08a2b0 con 0x7f21cc042f20 2021-03-09T09:45:54.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6542 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02bfe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:54.429 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22008 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:54.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc042f20 2021-03-09T09:45:54.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05b210 con 0x7f21cc042f20 2021-03-09T09:45:54.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc042f20 2021-03-09T09:45:54.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.425+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:54.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.429+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:54.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:54.429+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:45:59.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:59.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:59.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6542 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02bfe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:59.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6542 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02bfe0).stop 2021-03-09T09:45:59.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:59.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=CLOSED pgs=6542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:59.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:59.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:59.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:59.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:45:59.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053ba0 con 0x7f21c4003cb0 2021-03-09T09:45:59.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:45:59.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:45:59.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:59.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:45:59.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:59.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:59.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:59.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:59.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:59.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:59.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026730 con 0x7f21cc052490 2021-03-09T09:45:59.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04f8e0 con 0x7f21cc052490 2021-03-09T09:45:59.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6552 cs=0 l=1 rev1=1 rx=0x7f21dc05b5d0 tx=0x7f21dc0ab180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:59.434 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22042 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:45:59.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc09b220 con 0x7f21cc052490 2021-03-09T09:45:59.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc052490 2021-03-09T09:45:59.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00a130 con 0x7f21cc052490 2021-03-09T09:45:59.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.433+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:45:59.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.433+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:45:59.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:45:59.433+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:04.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:04.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:04.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6552 cs=0 l=1 rev1=1 rx=0x7f21dc05b5d0 tx=0x7f21dc0ab180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:04.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6552 cs=0 l=1 rev1=1 rx=0x7f21dc05b5d0 tx=0x7f21dc0ab180).stop 2021-03-09T09:46:04.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:04.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=CLOSED pgs=6552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:04.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:04.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:04.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:04.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc005e30 2021-03-09T09:46:04.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fc20 con 0x7f21c4003cb0 2021-03-09T09:46:04.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:46:04.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:04.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:04.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:04.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:04.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:04.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:04.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:04.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:04.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:04.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f4a0 con 0x7f21cc042f20 2021-03-09T09:46:04.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026730 con 0x7f21cc042f20 2021-03-09T09:46:04.439 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22082 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:04.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6564 cs=0 l=1 rev1=1 rx=0x7f21dc0982f0 tx=0x7f21dc098490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:04.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:46:04.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc054000 con 0x7f21cc042f20 2021-03-09T09:46:04.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.436+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:46:04.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.436+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:04.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.436+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:04.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:04.436+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:09.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:09.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:09.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6564 cs=0 l=1 rev1=1 rx=0x7f21dc0982f0 tx=0x7f21dc098490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:09.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6564 cs=0 l=1 rev1=1 rx=0x7f21dc0982f0 tx=0x7f21dc098490).stop 2021-03-09T09:46:09.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:09.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=CLOSED pgs=6564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:09.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:09.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:09.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:09.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043900 con 0x7f21cc005e30 2021-03-09T09:46:09.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fc20 con 0x7f21c4003cb0 2021-03-09T09:46:09.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265f0 con 0x7f21cc052490 2021-03-09T09:46:09.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:09.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:09.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:09.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:09.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.428+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:09.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:09.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:09.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:09.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:09.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026730 con 0x7f21cc052490 2021-03-09T09:46:09.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f4a0 con 0x7f21cc052490 2021-03-09T09:46:09.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6574 cs=0 l=1 rev1=1 rx=0x7f21dc05b460 tx=0x7f21dc015b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:09.442 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22116 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:09.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004100 con 0x7f21cc052490 2021-03-09T09:46:09.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:46:09.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.436+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00a130 con 0x7f21cc052490 2021-03-09T09:46:09.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.436+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:09.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.440+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:09.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:09.440+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:14.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:14.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:14.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6574 cs=0 l=1 rev1=1 rx=0x7f21dc05b460 tx=0x7f21dc015b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:14.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 secure :-1 s=READY pgs=6574 cs=0 l=1 rev1=1 rx=0x7f21dc05b460 tx=0x7f21dc015b00).stop 2021-03-09T09:46:14.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:14.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502c0 unknown :-1 s=CLOSED pgs=6574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:14.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:14.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:14.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:14.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b790 con 0x7f21cc005e30 2021-03-09T09:46:14.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0268c0 con 0x7f21c4003cb0 2021-03-09T09:46:14.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fc20 con 0x7f21cc042f20 2021-03-09T09:46:14.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:14.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:14.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:14.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:14.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:14.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:14.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:14.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:14.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:14.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:46:14.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026730 con 0x7f21cc042f20 2021-03-09T09:46:14.446 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:14.436+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22156 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:14.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6587 cs=0 l=1 rev1=1 rx=0x7f21cc02d8f0 tx=0x7f21dc008ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:14.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0781d0 con 0x7f21cc042f20 2021-03-09T09:46:14.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T09:46:14.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc042f20 2021-03-09T09:46:14.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.448+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:14.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.448+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:14.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:14.448+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:19.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:19.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:19.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6587 cs=0 l=1 rev1=1 rx=0x7f21cc02d8f0 tx=0x7f21dc008ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:19.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 secure :-1 s=READY pgs=6587 cs=0 l=1 rev1=1 rx=0x7f21cc02d8f0 tx=0x7f21dc008ed0).stop 2021-03-09T09:46:19.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0507b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:19.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0507b0 unknown :-1 s=CLOSED pgs=6587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:19.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:19.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:19.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:19.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:46:19.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fa20 con 0x7f21c4003cb0 2021-03-09T09:46:19.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04fb60 con 0x7f21cc052490 2021-03-09T09:46:19.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:19.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:19.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:19.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:19.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:19.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:19.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:19.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:19.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:19.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f3f0 con 0x7f21cc052490 2021-03-09T09:46:19.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023390 con 0x7f21cc052490 2021-03-09T09:46:19.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502a0 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 rx=0x7f21e4055340 tx=0x7f21e40552a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:19.451 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:19.440+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22190 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:19.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.444+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21cc052490 2021-03-09T09:46:19.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.444+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009890 con 0x7f21cc052490 2021-03-09T09:46:19.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21cc052490 2021-03-09T09:46:19.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.448+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:19.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.448+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:19.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:19.448+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:22.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:22.104 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:24.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:24.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:24.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502a0 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 rx=0x7f21e4055340 tx=0x7f21e40552a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:24.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502a0 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 rx=0x7f21e4055340 tx=0x7f21e40552a0).stop 2021-03-09T09:46:24.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0502a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:24.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0502a0 unknown :-1 s=CLOSED pgs=6595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:24.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:24.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:24.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:24.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T09:46:24.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027040 con 0x7f21c4003cb0 2021-03-09T09:46:24.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027180 con 0x7f21cc042f20 2021-03-09T09:46:24.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:24.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:24.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:24.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:24.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:24.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:24.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:24.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:24.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:24.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f580 con 0x7f21cc042f20 2021-03-09T09:46:24.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026730 con 0x7f21cc042f20 2021-03-09T09:46:24.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050790 secure :-1 s=READY pgs=6606 cs=0 l=1 rev1=1 rx=0x7f21dc092040 tx=0x7f21dc07cc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:24.455 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:24.444+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22230 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:24.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc098320 con 0x7f21cc042f20 2021-03-09T09:46:24.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00dea0 con 0x7f21cc042f20 2021-03-09T09:46:24.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:46:24.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.452+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:24.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.452+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:24.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:24.452+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:29.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:29.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:29.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050790 secure :-1 s=READY pgs=6606 cs=0 l=1 rev1=1 rx=0x7f21dc092040 tx=0x7f21dc07cc20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:29.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050790 secure :-1 s=READY pgs=6606 cs=0 l=1 rev1=1 rx=0x7f21dc092040 tx=0x7f21dc07cc20).stop 2021-03-09T09:46:29.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:29.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050790 unknown :-1 s=CLOSED pgs=6606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:29.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:29.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:29.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:29.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cf50 con 0x7f21cc005e30 2021-03-09T09:46:29.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0268d0 con 0x7f21c4003cb0 2021-03-09T09:46:29.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027190 con 0x7f21cc052490 2021-03-09T09:46:29.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:29.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:29.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:29.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:29.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:29.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:29.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:29.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:29.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:29.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d6d0 con 0x7f21cc052490 2021-03-09T09:46:29.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f3f0 con 0x7f21cc052490 2021-03-09T09:46:29.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6616 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40b2000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:29.460 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:29.448+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22264 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:29.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc052490 2021-03-09T09:46:29.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc052490 2021-03-09T09:46:29.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21cc052490 2021-03-09T09:46:29.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.456+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:29.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.456+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:29.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:29.456+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:34.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:46:34.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:34.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6616 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40b2000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:34.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6616 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40b2000).stop 2021-03-09T09:46:34.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:34.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:34.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:34.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:34.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:34.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cf50 con 0x7f21cc005e30 2021-03-09T09:46:34.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:34.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f80 con 0x7f21c4003cb0 2021-03-09T09:46:34.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023660 con 0x7f21cc042f20 2021-03-09T09:46:34.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:34.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:34.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:34.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:34.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:34.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:34.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:34.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:34.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0268d0 con 0x7f21cc042f20 2021-03-09T09:46:34.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d6d0 con 0x7f21cc042f20 2021-03-09T09:46:34.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6627 cs=0 l=1 rev1=1 rx=0x7f21e40159f0 tx=0x7f21e4031500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:34.463 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:34.456+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22304 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:34.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21cc042f20 2021-03-09T09:46:34.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a63f0 con 0x7f21cc042f20 2021-03-09T09:46:34.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.460+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T09:46:34.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.460+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:34.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.460+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:34.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:34.460+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:39.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:39.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:39.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6627 cs=0 l=1 rev1=1 rx=0x7f21e40159f0 tx=0x7f21e4031500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:39.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6627 cs=0 l=1 rev1=1 rx=0x7f21e40159f0 tx=0x7f21e4031500).stop 2021-03-09T09:46:39.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:39.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:39.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:39.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:39.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:39.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cf20 con 0x7f21cc005e30 2021-03-09T09:46:39.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d870 con 0x7f21c4003cb0 2021-03-09T09:46:39.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d270 con 0x7f21cc052490 2021-03-09T09:46:39.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:39.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:39.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:39.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:39.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.456+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:39.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:39.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:39.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:39.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:39.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f3f0 con 0x7f21cc052490 2021-03-09T09:46:39.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0268d0 con 0x7f21cc052490 2021-03-09T09:46:39.467 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22338 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:39.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6641 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e4024650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:39.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.460+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031340 con 0x7f21cc052490 2021-03-09T09:46:39.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a63f0 con 0x7f21cc052490 2021-03-09T09:46:39.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:46:39.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.468+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:39.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.468+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:39.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:39.468+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:44.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:44.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:44.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6641 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e4024650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:44.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6641 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e4024650).stop 2021-03-09T09:46:44.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:44.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:44.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:44.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:44.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:44.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054180 con 0x7f21cc005e30 2021-03-09T09:46:44.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0268d0 con 0x7f21c4003cb0 2021-03-09T09:46:44.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T09:46:44.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:44.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:44.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:44.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:44.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:44.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:44.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:44.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:44.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:44.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d6d0 con 0x7f21cc042f20 2021-03-09T09:46:44.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f3f0 con 0x7f21cc042f20 2021-03-09T09:46:44.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6648 cs=0 l=1 rev1=1 rx=0x7f21cc02d810 tx=0x7f21e40a5530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:44.472 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:44.464+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22378 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:44.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031340 con 0x7f21cc042f20 2021-03-09T09:46:44.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T09:46:44.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc042f20 2021-03-09T09:46:44.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.472+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:44.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.472+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:44.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:44.472+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:49.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:49.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:49.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6648 cs=0 l=1 rev1=1 rx=0x7f21cc02d810 tx=0x7f21e40a5530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:49.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6648 cs=0 l=1 rev1=1 rx=0x7f21cc02d810 tx=0x7f21e40a5530).stop 2021-03-09T09:46:49.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:49.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:49.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:49.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:49.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:49.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0154e0 con 0x7f21cc005e30 2021-03-09T09:46:49.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009730 con 0x7f21c4003cb0 2021-03-09T09:46:49.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04da20 con 0x7f21cc052490 2021-03-09T09:46:49.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:49.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:49.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:49.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:49.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:49.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:49.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:49.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:49.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:49.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f3f0 con 0x7f21cc052490 2021-03-09T09:46:49.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d6d0 con 0x7f21cc052490 2021-03-09T09:46:49.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6659 cs=0 l=1 rev1=1 rx=0x7f21cc042940 tx=0x7f21e4041290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:49.477 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22412 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:49.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc052490 2021-03-09T09:46:49.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4008e50 con 0x7f21cc052490 2021-03-09T09:46:49.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031260 con 0x7f21cc052490 2021-03-09T09:46:49.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.472+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:49.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.476+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:49.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:49.476+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:52.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:52.142 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:46:54.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:54.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:54.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6659 cs=0 l=1 rev1=1 rx=0x7f21cc042940 tx=0x7f21e4041290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:54.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6659 cs=0 l=1 rev1=1 rx=0x7f21cc042940 tx=0x7f21e4041290).stop 2021-03-09T09:46:54.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:54.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:54.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:54.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:54.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:54.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037710 con 0x7f21cc005e30 2021-03-09T09:46:54.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094fa0 con 0x7f21c4003cb0 2021-03-09T09:46:54.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d6d0 con 0x7f21cc042f20 2021-03-09T09:46:54.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:54.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:54.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:54.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:54.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.467+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:54.479 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22452 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:54.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:54.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:54.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0268d0 con 0x7f21cc042f20 2021-03-09T09:46:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f3f0 con 0x7f21cc042f20 2021-03-09T09:46:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.471+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6673 cs=0 l=1 rev1=1 rx=0x7f21e4008bb0 tx=0x7f21e40b1440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:54.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.475+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T09:46:54.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.475+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4048180 con 0x7f21cc042f20 2021-03-09T09:46:54.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21cc042f20 2021-03-09T09:46:54.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.479+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:54.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.479+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:54.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:54.479+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:46:59.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:46:59.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:59.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6673 cs=0 l=1 rev1=1 rx=0x7f21e4008bb0 tx=0x7f21e40b1440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:59.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6673 cs=0 l=1 rev1=1 rx=0x7f21e4008bb0 tx=0x7f21e40b1440).stop 2021-03-09T09:46:59.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:59.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:59.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:59.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:59.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:59.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042970 con 0x7f21cc005e30 2021-03-09T09:46:59.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f3f0 con 0x7f21c4003cb0 2021-03-09T09:46:59.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037570 con 0x7f21cc052490 2021-03-09T09:46:59.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:46:59.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:59.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:46:59.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:59.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:59.484 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22486 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:46:59.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:59.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:59.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:59.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:59.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0376b0 con 0x7f21cc052490 2021-03-09T09:46:59.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0268d0 con 0x7f21cc052490 2021-03-09T09:46:59.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.475+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6683 cs=0 l=1 rev1=1 rx=0x7f21e4018ec0 tx=0x7f21e40059e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:59.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005e70 con 0x7f21cc052490 2021-03-09T09:46:59.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc052490 2021-03-09T09:46:59.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.483+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc052490 2021-03-09T09:46:59.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.483+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:46:59.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.483+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:46:59.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:46:59.483+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:04.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:04.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:04.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6683 cs=0 l=1 rev1=1 rx=0x7f21e4018ec0 tx=0x7f21e40059e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:04.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6683 cs=0 l=1 rev1=1 rx=0x7f21e4018ec0 tx=0x7f21e40059e0).stop 2021-03-09T09:47:04.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc037570 con 0x7f21cc005e30 2021-03-09T09:47:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0268d0 con 0x7f21c4003cb0 2021-03-09T09:47:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f3f0 con 0x7f21cc042f20 2021-03-09T09:47:04.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:04.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:04.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:04.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:04.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.479+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:04.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:04.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:04.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:04.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:04.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d6d0 con 0x7f21cc042f20 2021-03-09T09:47:04.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0376b0 con 0x7f21cc042f20 2021-03-09T09:47:04.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6691 cs=0 l=1 rev1=1 rx=0x7f21e408b060 tx=0x7f21e4053070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:04.491 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:04.483+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22526 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:04.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.487+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005e70 con 0x7f21cc042f20 2021-03-09T09:47:04.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.487+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T09:47:04.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.491+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5300 con 0x7f21cc042f20 2021-03-09T09:47:04.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.491+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:04.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.495+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:04.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:04.495+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:09.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:09.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:09.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6691 cs=0 l=1 rev1=1 rx=0x7f21e408b060 tx=0x7f21e4053070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:09.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6691 cs=0 l=1 rev1=1 rx=0x7f21e408b060 tx=0x7f21e4053070).stop 2021-03-09T09:47:09.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:09.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:09.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:09.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:09.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:09.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f220 con 0x7f21cc005e30 2021-03-09T09:47:09.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f360 con 0x7f21c4003cb0 2021-03-09T09:47:09.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d8b0 con 0x7f21cc052490 2021-03-09T09:47:09.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:09.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:09.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:09.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.483+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:09.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:09.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:09.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:09.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:09.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:09.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03d9f0 con 0x7f21cc052490 2021-03-09T09:47:09.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f580 con 0x7f21cc052490 2021-03-09T09:47:09.495 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22561 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:09.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.487+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6702 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0b3200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:09.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.491+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:47:09.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.491+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc052490 2021-03-09T09:47:09.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.495+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:47:09.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.495+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:09.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.495+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:09.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:09.495+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:14.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:14.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:14.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6702 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0b3200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:14.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6702 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0b3200).stop 2021-03-09T09:47:14.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:14.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:14.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:14.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:14.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:14.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f1f0 con 0x7f21cc005e30 2021-03-09T09:47:14.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f330 con 0x7f21c4003cb0 2021-03-09T09:47:14.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f470 con 0x7f21cc042f20 2021-03-09T09:47:14.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:14.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.487+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:14.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:14.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:14.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:14.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:14.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:14.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:14.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:14.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f5b0 con 0x7f21cc042f20 2021-03-09T09:47:14.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03d9f0 con 0x7f21cc042f20 2021-03-09T09:47:14.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6712 cs=0 l=1 rev1=1 rx=0x7f21dc028580 tx=0x7f21dc0281e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:14.498 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:14.491+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22600 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:14.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.495+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T09:47:14.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06b840 con 0x7f21cc042f20 2021-03-09T09:47:14.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc077fc0 con 0x7f21cc042f20 2021-03-09T09:47:14.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.499+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:14.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.499+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:14.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:14.499+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:19.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:19.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:19.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6712 cs=0 l=1 rev1=1 rx=0x7f21dc028580 tx=0x7f21dc0281e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:19.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6712 cs=0 l=1 rev1=1 rx=0x7f21dc028580 tx=0x7f21dc0281e0).stop 2021-03-09T09:47:19.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:19.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:19.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:19.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.491+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:19.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:19.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f740 con 0x7f21cc005e30 2021-03-09T09:47:19.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f1a0 con 0x7f21c4003cb0 2021-03-09T09:47:19.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f2e0 con 0x7f21cc052490 2021-03-09T09:47:19.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:19.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:19.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:19.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:19.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:19.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:19.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:19.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:19.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:19.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f420 con 0x7f21cc052490 2021-03-09T09:47:19.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d6d0 con 0x7f21cc052490 2021-03-09T09:47:19.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6723 cs=0 l=1 rev1=1 rx=0x7f21cc01af70 tx=0x7f21e409bda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:19.503 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:19.495+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22634 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:19.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003af0 con 0x7f21cc052490 2021-03-09T09:47:19.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404a0d0 con 0x7f21cc052490 2021-03-09T09:47:19.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21cc052490 2021-03-09T09:47:19.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.503+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:19.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.503+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:19.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:19.503+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:22.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:22.190 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:24.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:24.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:24.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6723 cs=0 l=1 rev1=1 rx=0x7f21cc01af70 tx=0x7f21e409bda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:24.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6723 cs=0 l=1 rev1=1 rx=0x7f21cc01af70 tx=0x7f21e409bda0).stop 2021-03-09T09:47:24.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:24.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:24.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:24.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:24.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:24.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T09:47:24.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052870 con 0x7f21c4003cb0 2021-03-09T09:47:24.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f740 con 0x7f21cc042f20 2021-03-09T09:47:24.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:24.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:24.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:24.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:24.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:24.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:24.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:24.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:24.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:24.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04da10 con 0x7f21cc042f20 2021-03-09T09:47:24.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.499+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f5b0 con 0x7f21cc042f20 2021-03-09T09:47:24.508 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:24.503+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22674 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:24.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6734 cs=0 l=1 rev1=1 rx=0x7f21cc094690 tx=0x7f21dc085230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:24.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052e10 con 0x7f21cc042f20 2021-03-09T09:47:24.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052c30 con 0x7f21cc042f20 2021-03-09T09:47:24.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc042f20 2021-03-09T09:47:24.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.507+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:24.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.507+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:24.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:24.507+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:29.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:29.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:29.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6734 cs=0 l=1 rev1=1 rx=0x7f21cc094690 tx=0x7f21dc085230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:29.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6734 cs=0 l=1 rev1=1 rx=0x7f21cc094690 tx=0x7f21dc085230).stop 2021-03-09T09:47:29.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:29.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:29.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:29.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:29.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:29.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f750 con 0x7f21cc005e30 2021-03-09T09:47:29.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b330 con 0x7f21c4003cb0 2021-03-09T09:47:29.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b470 con 0x7f21cc052490 2021-03-09T09:47:29.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:29.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:29.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:29.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:29.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:29.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:29.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:29.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:29.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:29.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b5b0 con 0x7f21cc052490 2021-03-09T09:47:29.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04da10 con 0x7f21cc052490 2021-03-09T09:47:29.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6744 cs=0 l=1 rev1=1 rx=0x7f21dc056630 tx=0x7f21dc0193f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:29.513 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:29.507+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22708 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:29.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052e10 con 0x7f21cc052490 2021-03-09T09:47:29.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc052490 2021-03-09T09:47:29.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc005920 con 0x7f21cc052490 2021-03-09T09:47:29.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.511+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:29.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.511+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:29.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:29.511+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:34.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:34.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:34.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6744 cs=0 l=1 rev1=1 rx=0x7f21dc056630 tx=0x7f21dc0193f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:34.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6744 cs=0 l=1 rev1=1 rx=0x7f21dc056630 tx=0x7f21dc0193f0).stop 2021-03-09T09:47:34.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:34.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:34.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:34.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:34.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:34.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f750 con 0x7f21cc005e30 2021-03-09T09:47:34.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a7b0 con 0x7f21c4003cb0 2021-03-09T09:47:34.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04da10 con 0x7f21cc042f20 2021-03-09T09:47:34.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:34.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:34.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:34.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:34.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:34.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:34.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:34.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:34.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:34.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015830 con 0x7f21cc042f20 2021-03-09T09:47:34.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b5b0 con 0x7f21cc042f20 2021-03-09T09:47:34.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.507+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6755 cs=0 l=1 rev1=1 rx=0x7f21cc009800 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:34.516 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:34.511+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22747 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:34.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:47:34.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc021390 con 0x7f21cc042f20 2021-03-09T09:47:34.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21cc042f20 2021-03-09T09:47:34.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.515+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:34.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:34.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:34.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:39.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:39.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:39.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6755 cs=0 l=1 rev1=1 rx=0x7f21cc009800 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:39.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6755 cs=0 l=1 rev1=1 rx=0x7f21cc009800 tx=0x7f21dc010d20).stop 2021-03-09T09:47:39.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:39.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:39.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:39.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:39.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:39.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cd0 con 0x7f21cc005e30 2021-03-09T09:47:39.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d6d0 con 0x7f21c4003cb0 2021-03-09T09:47:39.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d810 con 0x7f21cc052490 2021-03-09T09:47:39.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:39.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:39.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:39.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:39.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:39.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:39.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:39.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:39.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:39.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d980 con 0x7f21cc052490 2021-03-09T09:47:39.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f420 con 0x7f21cc052490 2021-03-09T09:47:39.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6766 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4003bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:39.524 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:39.515+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:39.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404a0e0 con 0x7f21cc052490 2021-03-09T09:47:39.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:47:39.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:47:39.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.523+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:39.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.523+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:39.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:39.523+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:44.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:44.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:44.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6766 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4003bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:44.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6766 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4003bc0).stop 2021-03-09T09:47:44.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:44.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:44.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:44.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:44.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:44.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cd0 con 0x7f21cc005e30 2021-03-09T09:47:44.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f600 con 0x7f21c4003cb0 2021-03-09T09:47:44.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f5a0 con 0x7f21cc042f20 2021-03-09T09:47:44.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:44.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:44.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:44.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:44.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.515+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:44.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:44.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:44.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:44.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:44.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02f710 con 0x7f21cc042f20 2021-03-09T09:47:44.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015830 con 0x7f21cc042f20 2021-03-09T09:47:44.525 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22821 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:44.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6776 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc08a1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:44.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc042f20 2021-03-09T09:47:44.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0726e0 con 0x7f21cc042f20 2021-03-09T09:47:44.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21cc042f20 2021-03-09T09:47:44.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.527+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:44.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.527+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:44.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:44.527+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:49.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:49.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:49.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6776 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc08a1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:49.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6776 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc08a1d0).stop 2021-03-09T09:47:49.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:49.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:49.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:49.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:49.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:49.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cd0 con 0x7f21cc005e30 2021-03-09T09:47:49.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0227b0 con 0x7f21c4003cb0 2021-03-09T09:47:49.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0228f0 con 0x7f21cc052490 2021-03-09T09:47:49.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:49.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:49.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:49.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:49.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.518+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:49.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:49.527 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22856 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:49.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:49.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:49.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:49.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d860 con 0x7f21cc052490 2021-03-09T09:47:49.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02f710 con 0x7f21cc052490 2021-03-09T09:47:49.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 rx=0x7f21cc042460 tx=0x7f21dc072750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:49.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:47:49.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21cc052490 2021-03-09T09:47:49.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0280b0 con 0x7f21cc052490 2021-03-09T09:47:49.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.530+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:49.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.530+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:49.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:49.530+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:52.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:52.237 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:47:54.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:54.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:54.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 rx=0x7f21cc042460 tx=0x7f21dc072750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:54.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 rx=0x7f21cc042460 tx=0x7f21dc072750).stop 2021-03-09T09:47:54.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:54.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:54.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:54.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:54.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:54.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:54.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:54.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022900 con 0x7f21cc005e30 2021-03-09T09:47:54.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cd0 con 0x7f21c4003cb0 2021-03-09T09:47:54.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:54.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d5c0 con 0x7f21cc042f20 2021-03-09T09:47:54.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:54.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:54.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:54.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:54.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:54.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:54.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d9f0 con 0x7f21cc042f20 2021-03-09T09:47:54.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d860 con 0x7f21cc042f20 2021-03-09T09:47:54.532 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22895 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:54.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6800 cs=0 l=1 rev1=1 rx=0x7f21cc0393e0 tx=0x7f21dc028620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:54.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc029350 con 0x7f21cc042f20 2021-03-09T09:47:54.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc042f20 2021-03-09T09:47:54.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc042f20 2021-03-09T09:47:54.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.530+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:54.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.534+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:54.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:54.534+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:47:59.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:59.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:59.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6800 cs=0 l=1 rev1=1 rx=0x7f21cc0393e0 tx=0x7f21dc028620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:59.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6800 cs=0 l=1 rev1=1 rx=0x7f21cc0393e0 tx=0x7f21dc028620).stop 2021-03-09T09:47:59.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:59.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:59.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:59.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:59.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:59.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d860 con 0x7f21cc005e30 2021-03-09T09:47:59.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047080 con 0x7f21c4003cb0 2021-03-09T09:47:59.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022740 con 0x7f21cc052490 2021-03-09T09:47:59.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:47:59.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:59.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:47:59.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:59.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.526+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:59.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:59.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:59.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:59.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:59.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0228b0 con 0x7f21cc052490 2021-03-09T09:47:59.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d980 con 0x7f21cc052490 2021-03-09T09:47:59.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6808 cs=0 l=1 rev1=1 rx=0x7f21cc01a900 tx=0x7f21e4003860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:59.537 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22930 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:47:59.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009430 con 0x7f21cc052490 2021-03-09T09:47:59.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40923f0 con 0x7f21cc052490 2021-03-09T09:47:59.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc052490 2021-03-09T09:47:59.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.534+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:47:59.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.538+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:47:59.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:47:59.538+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:04.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:04.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:04.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6808 cs=0 l=1 rev1=1 rx=0x7f21cc01a900 tx=0x7f21e4003860).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:04.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6808 cs=0 l=1 rev1=1 rx=0x7f21cc01a900 tx=0x7f21e4003860).stop 2021-03-09T09:48:04.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:04.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:04.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:04.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:04.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:04.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d860 con 0x7f21cc005e30 2021-03-09T09:48:04.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046ee0 con 0x7f21c4003cb0 2021-03-09T09:48:04.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047020 con 0x7f21cc042f20 2021-03-09T09:48:04.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:04.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:04.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:04.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:04.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:04.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:04.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:04.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:04.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:04.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047160 con 0x7f21cc042f20 2021-03-09T09:48:04.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d9f0 con 0x7f21cc042f20 2021-03-09T09:48:04.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6819 cs=0 l=1 rev1=1 rx=0x7f21cc02d2b0 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:04.541 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22969 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:04.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3140 con 0x7f21cc042f20 2021-03-09T09:48:04.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc042f20 2021-03-09T09:48:04.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:48:04.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.542+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:04.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.542+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:04.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:04.542+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:09.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:09.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:09.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6819 cs=0 l=1 rev1=1 rx=0x7f21cc02d2b0 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:09.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6819 cs=0 l=1 rev1=1 rx=0x7f21cc02d2b0 tx=0x7f21dc010d20).stop 2021-03-09T09:48:09.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:09.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:09.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:09.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:09.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:09.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d860 con 0x7f21cc005e30 2021-03-09T09:48:09.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d9a0 con 0x7f21c4003cb0 2021-03-09T09:48:09.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dae0 con 0x7f21cc052490 2021-03-09T09:48:09.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:09.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:09.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:09.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.534+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:09.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:09.543 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23004 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:09.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:09.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:09.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:09.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:09.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dc50 con 0x7f21cc052490 2021-03-09T09:48:09.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc047160 con 0x7f21cc052490 2021-03-09T09:48:09.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc055880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:09.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:48:09.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:48:09.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc052490 2021-03-09T09:48:09.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.546+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:09.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.550+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:09.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:09.550+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:14.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:14.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:14.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc055880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:14.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 rx=0x7f21dc0098b0 tx=0x7f21dc055880).stop 2021-03-09T09:48:14.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:14.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:14.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:14.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:14.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:14.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01df80 con 0x7f21cc005e30 2021-03-09T09:48:14.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057a20 con 0x7f21c4003cb0 2021-03-09T09:48:14.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7e0 con 0x7f21cc042f20 2021-03-09T09:48:14.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:14.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:14.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:14.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:14.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:14.548 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23043 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:14.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:14.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:14.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:14.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:14.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d950 con 0x7f21cc042f20 2021-03-09T09:48:14.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01dc50 con 0x7f21cc042f20 2021-03-09T09:48:14.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6845 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc0559d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:14.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc096810 con 0x7f21cc042f20 2021-03-09T09:48:14.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc055470 con 0x7f21cc042f20 2021-03-09T09:48:14.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21cc042f20 2021-03-09T09:48:14.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.554+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:14.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.554+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:14.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:14.554+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:19.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:19.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:19.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6845 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc0559d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:19.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6845 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc0559d0).stop 2021-03-09T09:48:19.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:19.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:19.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:19.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:19.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:19.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7b0 con 0x7f21cc005e30 2021-03-09T09:48:19.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dae0 con 0x7f21c4003cb0 2021-03-09T09:48:19.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dc20 con 0x7f21cc052490 2021-03-09T09:48:19.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:19.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:19.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:19.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:19.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:19.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:19.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:19.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:19.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:19.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dd60 con 0x7f21cc052490 2021-03-09T09:48:19.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d950 con 0x7f21cc052490 2021-03-09T09:48:19.555 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:19.546+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23078 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:19.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.550+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6852 cs=0 l=1 rev1=1 rx=0x7f21dc062d40 tx=0x7f21dc010e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:19.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0723b0 con 0x7f21cc052490 2021-03-09T09:48:19.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc096810 con 0x7f21cc052490 2021-03-09T09:48:19.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21cc052490 2021-03-09T09:48:19.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.554+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:19.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.558+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:19.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:19.558+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:22.281 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:22.284 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:24.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:24.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:24.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6852 cs=0 l=1 rev1=1 rx=0x7f21dc062d40 tx=0x7f21dc010e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:24.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6852 cs=0 l=1 rev1=1 rx=0x7f21dc062d40 tx=0x7f21dc010e00).stop 2021-03-09T09:48:24.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:24.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:24.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:24.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:24.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:24.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0862a0 con 0x7f21cc005e30 2021-03-09T09:48:24.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008340 con 0x7f21c4003cb0 2021-03-09T09:48:24.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d780 con 0x7f21cc042f20 2021-03-09T09:48:24.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:24.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:24.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:24.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:24.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:24.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:24.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:24.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:24.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:24.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d8f0 con 0x7f21cc042f20 2021-03-09T09:48:24.558 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23117 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:24.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.550+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0228b0 con 0x7f21cc042f20 2021-03-09T09:48:24.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.554+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6863 cs=0 l=1 rev1=1 rx=0x7f21e4016c20 tx=0x7f21e4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:24.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40093b0 con 0x7f21cc042f20 2021-03-09T09:48:24.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T09:48:24.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T09:48:24.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.558+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:24.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.562+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:24.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:24.562+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:29.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:29.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:29.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6863 cs=0 l=1 rev1=1 rx=0x7f21e4016c20 tx=0x7f21e4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:29.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 secure :-1 s=READY pgs=6863 cs=0 l=1 rev1=1 rx=0x7f21e4016c20 tx=0x7f21e4003ad0).stop 2021-03-09T09:48:29.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc050770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:29.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc050770 unknown :-1 s=CLOSED pgs=6863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:29.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d780 con 0x7f21cc005e30 2021-03-09T09:48:29.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086100 con 0x7f21c4003cb0 2021-03-09T09:48:29.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086240 con 0x7f21cc052490 2021-03-09T09:48:29.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:29.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:29.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:29.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:29.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:29.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:29.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:29.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:29.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:29.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01adf0 con 0x7f21cc052490 2021-03-09T09:48:29.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01dd60 con 0x7f21cc052490 2021-03-09T09:48:29.563 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23152 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:29.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.554+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6873 cs=0 l=1 rev1=1 rx=0x7f21dc064370 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:29.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc052490 2021-03-09T09:48:29.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0641d0 con 0x7f21cc052490 2021-03-09T09:48:29.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.562+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018490 con 0x7f21cc052490 2021-03-09T09:48:29.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.562+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:29.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.562+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:29.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:29.562+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:34.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:34.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:34.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6873 cs=0 l=1 rev1=1 rx=0x7f21dc064370 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:34.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6873 cs=0 l=1 rev1=1 rx=0x7f21dc064370 tx=0x7f21dc02dfc0).stop 2021-03-09T09:48:34.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:34.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:34.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:34.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc086100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:34.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:34.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:34.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:34.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d780 con 0x7f21cc005e30 2021-03-09T09:48:34.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b120 con 0x7f21c4003cb0 2021-03-09T09:48:34.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050770 con 0x7f21cc042f20 2021-03-09T09:48:34.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:34.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:34.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc086100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:34.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:34.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:34.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:34.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:34.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0508b0 con 0x7f21cc042f20 2021-03-09T09:48:34.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d8f0 con 0x7f21cc042f20 2021-03-09T09:48:34.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc086100 secure :-1 s=READY pgs=6883 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40173f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:34.567 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:34.558+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23191 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:34.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.562+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T09:48:34.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.562+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21cc042f20 2021-03-09T09:48:34.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.566+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T09:48:34.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.566+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:34.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.566+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:34.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:34.566+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:39.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc086100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:39.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc086100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:39.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc086100 secure :-1 s=READY pgs=6883 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40173f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:39.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc086100 secure :-1 s=READY pgs=6883 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40173f0).stop 2021-03-09T09:48:39.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc086100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:39.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc086100 unknown :-1 s=CLOSED pgs=6883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:39.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:39.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:39.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:39.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d780 con 0x7f21cc005e30 2021-03-09T09:48:39.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050770 con 0x7f21c4003cb0 2021-03-09T09:48:39.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d750 con 0x7f21cc052490 2021-03-09T09:48:39.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:39.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:39.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:39.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:39.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:39.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:39.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:39.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:39.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:39.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d8f0 con 0x7f21cc052490 2021-03-09T09:48:39.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01adf0 con 0x7f21cc052490 2021-03-09T09:48:39.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.561+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6894 cs=0 l=1 rev1=1 rx=0x7f21dc00d560 tx=0x7f21dc08a190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:39.571 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:39.565+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23226 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:39.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00d760 con 0x7f21cc052490 2021-03-09T09:48:39.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050eb0 con 0x7f21cc052490 2021-03-09T09:48:39.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:48:39.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.569+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:39.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:39.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:39.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:44.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:44.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:44.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6894 cs=0 l=1 rev1=1 rx=0x7f21dc00d560 tx=0x7f21dc08a190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:44.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6894 cs=0 l=1 rev1=1 rx=0x7f21dc00d560 tx=0x7f21dc08a190).stop 2021-03-09T09:48:44.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:44.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:44.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:44.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:44.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:44.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f260 con 0x7f21cc005e30 2021-03-09T09:48:44.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d750 con 0x7f21c4003cb0 2021-03-09T09:48:44.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0860c0 con 0x7f21cc042f20 2021-03-09T09:48:44.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:44.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:44.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:44.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:44.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:44.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:44.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:44.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:44.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:44.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086200 con 0x7f21cc042f20 2021-03-09T09:48:44.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.565+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d8f0 con 0x7f21cc042f20 2021-03-09T09:48:44.575 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:44.569+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23265 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:44.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ada0 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 rx=0x7f21cc042de0 tx=0x7f21dc057470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:44.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc021390 con 0x7f21cc042f20 2021-03-09T09:48:44.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:48:44.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21cc042f20 2021-03-09T09:48:44.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.573+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:44.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:44.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:44.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:49.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:49.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:49.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ada0 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 rx=0x7f21cc042de0 tx=0x7f21dc057470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:49.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ada0 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 rx=0x7f21cc042de0 tx=0x7f21dc057470).stop 2021-03-09T09:48:49.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01ada0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:49.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ada0 unknown :-1 s=CLOSED pgs=6904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:49.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d540 con 0x7f21cc005e30 2021-03-09T09:48:49.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0860c0 con 0x7f21c4003cb0 2021-03-09T09:48:49.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086390 con 0x7f21cc052490 2021-03-09T09:48:49.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:49.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:49.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:49.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:49.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:49.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:49.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:49.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:49.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:49.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0864d0 con 0x7f21cc052490 2021-03-09T09:48:49.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086200 con 0x7f21cc052490 2021-03-09T09:48:49.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.573+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6915 cs=0 l=1 rev1=1 rx=0x7f21dc096cb0 tx=0x7f21dc02bc10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:49.581 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:49.573+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:49.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc052490 2021-03-09T09:48:49.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc052490 2021-03-09T09:48:49.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.581+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02bd60 con 0x7f21cc052490 2021-03-09T09:48:49.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.581+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:49.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.581+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:49.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:49.581+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:52.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:52.359 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:48:54.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:54.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6915 cs=0 l=1 rev1=1 rx=0x7f21dc096cb0 tx=0x7f21dc02bc10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6915 cs=0 l=1 rev1=1 rx=0x7f21dc096cb0 tx=0x7f21dc02bc10).stop 2021-03-09T09:48:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d540 con 0x7f21cc005e30 2021-03-09T09:48:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e680 con 0x7f21c4003cb0 2021-03-09T09:48:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e7c0 con 0x7f21cc042f20 2021-03-09T09:48:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:54.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:54.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.573+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:54.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0860c0 con 0x7f21cc042f20 2021-03-09T09:48:54.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0864d0 con 0x7f21cc042f20 2021-03-09T09:48:54.587 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23339 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:54.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01acf0 secure :-1 s=READY pgs=6926 cs=0 l=1 rev1=1 rx=0x7f21dc0bb710 tx=0x7f21dc0640f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:54.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.581+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:48:54.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.581+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21cc042f20 2021-03-09T09:48:54.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc096de0 con 0x7f21cc042f20 2021-03-09T09:48:54.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.589+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:54.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.589+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:54.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:54.589+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:48:59.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:59.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:59.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01acf0 secure :-1 s=READY pgs=6926 cs=0 l=1 rev1=1 rx=0x7f21dc0bb710 tx=0x7f21dc0640f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:59.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01acf0 secure :-1 s=READY pgs=6926 cs=0 l=1 rev1=1 rx=0x7f21dc0bb710 tx=0x7f21dc0640f0).stop 2021-03-09T09:48:59.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01acf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:59.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01acf0 unknown :-1 s=CLOSED pgs=6926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:59.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:59.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:59.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:59.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d540 con 0x7f21cc005e30 2021-03-09T09:48:59.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086250 con 0x7f21c4003cb0 2021-03-09T09:48:59.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086390 con 0x7f21cc052490 2021-03-09T09:48:59.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:48:59.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:59.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:59.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:48:59.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:59.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:59.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:59.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:59.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:59.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086500 con 0x7f21cc052490 2021-03-09T09:48:59.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0508b0 con 0x7f21cc052490 2021-03-09T09:48:59.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6936 cs=0 l=1 rev1=1 rx=0x7f21e408a730 tx=0x7f21e408a590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:59.589 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23374 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:48:59.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408a430 con 0x7f21cc052490 2021-03-09T09:48:59.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.581+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21cc052490 2021-03-09T09:48:59.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40092b0 con 0x7f21cc052490 2021-03-09T09:48:59.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.585+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:48:59.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.589+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:48:59.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:48:59.589+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:04.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:04.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:04.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6936 cs=0 l=1 rev1=1 rx=0x7f21e408a730 tx=0x7f21e408a590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:04.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6936 cs=0 l=1 rev1=1 rx=0x7f21e408a730 tx=0x7f21e408a590).stop 2021-03-09T09:49:04.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:04.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:04.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:04.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:04.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:04.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d540 con 0x7f21cc005e30 2021-03-09T09:49:04.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046790 con 0x7f21c4003cb0 2021-03-09T09:49:04.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013230 con 0x7f21cc042f20 2021-03-09T09:49:04.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:04.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:04.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:04.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:04.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:04.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:04.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:04.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:04.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:04.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013370 con 0x7f21cc042f20 2021-03-09T09:49:04.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0860c0 con 0x7f21cc042f20 2021-03-09T09:49:04.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ac50 secure :-1 s=READY pgs=6947 cs=0 l=1 rev1=1 rx=0x7f21dc072340 tx=0x7f21dc064080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:04.593 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:04.585+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23413 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:04.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06a430 con 0x7f21cc042f20 2021-03-09T09:49:04.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21cc042f20 2021-03-09T09:49:04.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.593+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc096de0 con 0x7f21cc042f20 2021-03-09T09:49:04.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.593+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:04.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.593+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:04.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:04.593+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:09.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:09.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:09.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ac50 secure :-1 s=READY pgs=6947 cs=0 l=1 rev1=1 rx=0x7f21dc072340 tx=0x7f21dc064080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:09.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ac50 secure :-1 s=READY pgs=6947 cs=0 l=1 rev1=1 rx=0x7f21dc072340 tx=0x7f21dc064080).stop 2021-03-09T09:49:09.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01ac50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:09.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01ac50 unknown :-1 s=CLOSED pgs=6947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:09.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:09.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:09.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:09.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d540 con 0x7f21cc005e30 2021-03-09T09:49:09.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094db0 con 0x7f21c4003cb0 2021-03-09T09:49:09.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013230 con 0x7f21cc052490 2021-03-09T09:49:09.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:09.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:09.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:09.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:09.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:09.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:09.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:09.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:09.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:09.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0860c0 con 0x7f21cc052490 2021-03-09T09:49:09.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013370 con 0x7f21cc052490 2021-03-09T09:49:09.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6958 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc029f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:09.602 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23448 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:09.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc052490 2021-03-09T09:49:09.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.593+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065610 con 0x7f21cc052490 2021-03-09T09:49:09.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:49:09.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.597+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:09.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.597+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:09.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:09.597+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:14.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:14.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:14.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6958 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc029f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:14.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 secure :-1 s=READY pgs=6958 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc029f30).stop 2021-03-09T09:49:14.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc050280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:14.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc050280 unknown :-1 s=CLOSED pgs=6958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:14.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:14.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:14.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:14.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:49:14.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094bf0 con 0x7f21c4003cb0 2021-03-09T09:49:14.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094d30 con 0x7f21cc042f20 2021-03-09T09:49:14.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:14.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:14.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:14.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:14.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:14.601 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:14.593+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23487 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:14.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:14.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:14.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:14.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:14.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044c60 con 0x7f21cc042f20 2021-03-09T09:49:14.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0860c0 con 0x7f21cc042f20 2021-03-09T09:49:14.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aba0 secure :-1 s=READY pgs=6972 cs=0 l=1 rev1=1 rx=0x7f21dc0655a0 tx=0x7f21dc032ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:14.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc042f20 2021-03-09T09:49:14.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21cc042f20 2021-03-09T09:49:14.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:49:14.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.605+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:14.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.605+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:14.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:14.605+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:19.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:19.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:19.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aba0 secure :-1 s=READY pgs=6972 cs=0 l=1 rev1=1 rx=0x7f21dc0655a0 tx=0x7f21dc032ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:19.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aba0 secure :-1 s=READY pgs=6972 cs=0 l=1 rev1=1 rx=0x7f21dc0655a0 tx=0x7f21dc032ee0).stop 2021-03-09T09:49:19.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01aba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:19.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aba0 unknown :-1 s=CLOSED pgs=6972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:19.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:19.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:19.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:19.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044f90 con 0x7f21cc005e30 2021-03-09T09:49:19.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21c4003cb0 2021-03-09T09:49:19.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc052490 2021-03-09T09:49:19.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:19.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:19.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:19.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:19.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:19.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:19.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:19.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:19.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:19.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc085f80 con 0x7f21cc052490 2021-03-09T09:49:19.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044c60 con 0x7f21cc052490 2021-03-09T09:49:19.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094a20 secure :-1 s=READY pgs=6979 cs=0 l=1 rev1=1 rx=0x7f21dc033070 tx=0x7f21dc00db60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:19.607 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:19.601+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23519 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:19.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21cc052490 2021-03-09T09:49:19.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc052490 2021-03-09T09:49:19.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018700 con 0x7f21cc052490 2021-03-09T09:49:19.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.605+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:19.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.605+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:19.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:19.605+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:22.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:22.406 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:24.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:24.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:24.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094a20 secure :-1 s=READY pgs=6979 cs=0 l=1 rev1=1 rx=0x7f21dc033070 tx=0x7f21dc00db60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:24.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094a20 secure :-1 s=READY pgs=6979 cs=0 l=1 rev1=1 rx=0x7f21dc033070 tx=0x7f21dc00db60).stop 2021-03-09T09:49:24.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:24.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094a20 unknown :-1 s=CLOSED pgs=6979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:24.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:24.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:24.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:24.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d720 con 0x7f21cc005e30 2021-03-09T09:49:24.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21c4003cb0 2021-03-09T09:49:24.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ab00 con 0x7f21cc042f20 2021-03-09T09:49:24.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:24.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:24.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:24.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:24.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:24.610 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23561 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:24.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:24.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:24.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:24.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:24.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ac40 con 0x7f21cc042f20 2021-03-09T09:49:24.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc085f80 con 0x7f21cc042f20 2021-03-09T09:49:24.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044bb0 secure :-1 s=READY pgs=6993 cs=0 l=1 rev1=1 rx=0x7f21dc00c780 tx=0x7f21dc085ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:24.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc091070 con 0x7f21cc042f20 2021-03-09T09:49:24.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09ae10 con 0x7f21cc042f20 2021-03-09T09:49:24.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.609+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018700 con 0x7f21cc042f20 2021-03-09T09:49:24.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.609+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:24.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.613+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:24.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:24.613+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:29.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc044bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:29.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc044bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:29.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044bb0 secure :-1 s=READY pgs=6993 cs=0 l=1 rev1=1 rx=0x7f21dc00c780 tx=0x7f21dc085ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:29.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044bb0 secure :-1 s=READY pgs=6993 cs=0 l=1 rev1=1 rx=0x7f21dc00c780 tx=0x7f21dc085ef0).stop 2021-03-09T09:49:29.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc044bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:29.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044bb0 unknown :-1 s=CLOSED pgs=6993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:29.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:29.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:29.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:29.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:49:29.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01aab0 con 0x7f21c4003cb0 2021-03-09T09:49:29.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094d20 con 0x7f21cc052490 2021-03-09T09:49:29.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:29.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:29.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:29.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:29.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.605+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:29.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:29.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:29.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:29.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:29.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc085f30 con 0x7f21cc052490 2021-03-09T09:49:29.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086500 con 0x7f21cc052490 2021-03-09T09:49:29.616 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23596 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:29.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.609+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094830 secure :-1 s=READY pgs=7000 cs=0 l=1 rev1=1 rx=0x7f21e407f210 tx=0x7f21e40b1280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:29.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.613+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21cc052490 2021-03-09T09:49:29.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.613+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21cc052490 2021-03-09T09:49:29.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.617+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21cc052490 2021-03-09T09:49:29.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.617+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:29.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.617+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:29.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:29.617+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:34.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:34.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:34.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094830 secure :-1 s=READY pgs=7000 cs=0 l=1 rev1=1 rx=0x7f21e407f210 tx=0x7f21e40b1280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:34.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094830 secure :-1 s=READY pgs=7000 cs=0 l=1 rev1=1 rx=0x7f21e407f210 tx=0x7f21e40b1280).stop 2021-03-09T09:49:34.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:34.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094830 unknown :-1 s=CLOSED pgs=7000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:34.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:34.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:34.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:34.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094d20 con 0x7f21cc005e30 2021-03-09T09:49:34.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21c4003cb0 2021-03-09T09:49:34.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01add0 con 0x7f21cc042f20 2021-03-09T09:49:34.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:34.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:34.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:34.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:34.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:34.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:34.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:34.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:34.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:34.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b570 con 0x7f21cc042f20 2021-03-09T09:49:34.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ac40 con 0x7f21cc042f20 2021-03-09T09:49:34.621 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23635 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:34.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044b40 secure :-1 s=READY pgs=7012 cs=0 l=1 rev1=1 rx=0x7f21cc01b2b0 tx=0x7f21dc029ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:34.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.616+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bb5a0 con 0x7f21cc042f20 2021-03-09T09:49:34.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.616+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc042f20 2021-03-09T09:49:34.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.620+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:49:34.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.620+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:34.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.624+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:34.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:34.624+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:39.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc044b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:39.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc044b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:39.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044b40 secure :-1 s=READY pgs=7012 cs=0 l=1 rev1=1 rx=0x7f21cc01b2b0 tx=0x7f21dc029ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:39.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044b40 secure :-1 s=READY pgs=7012 cs=0 l=1 rev1=1 rx=0x7f21cc01b2b0 tx=0x7f21dc029ec0).stop 2021-03-09T09:49:39.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc044b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:39.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc044b40 unknown :-1 s=CLOSED pgs=7012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:39.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:39.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:39.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0944f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:39.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:49:39.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085d40 con 0x7f21c4003cb0 2021-03-09T09:49:39.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0949e0 con 0x7f21cc052490 2021-03-09T09:49:39.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:39.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:39.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:39.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:39.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.612+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:39.623 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23669 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:39.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:39.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:39.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:39.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:39.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094b50 con 0x7f21cc052490 2021-03-09T09:49:39.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b570 con 0x7f21cc052490 2021-03-09T09:49:39.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0944f0 secure :-1 s=READY pgs=7025 cs=0 l=1 rev1=1 rx=0x7f21dc081fe0 tx=0x7f21dc052c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:39.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.620+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc057ea0 con 0x7f21cc052490 2021-03-09T09:49:39.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc057ea0 con 0x7f21cc052490 2021-03-09T09:49:39.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:49:39.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.628+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:39.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.628+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:39.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:39.628+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:44.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0944f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:49:44.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0944f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:44.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0944f0 secure :-1 s=READY pgs=7025 cs=0 l=1 rev1=1 rx=0x7f21dc081fe0 tx=0x7f21dc052c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:44.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0944f0 secure :-1 s=READY pgs=7025 cs=0 l=1 rev1=1 rx=0x7f21dc081fe0 tx=0x7f21dc052c30).stop 2021-03-09T09:49:44.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0944f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:44.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0944f0 unknown :-1 s=CLOSED pgs=7025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:44.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:44.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:44.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:44.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f920 con 0x7f21cc005e30 2021-03-09T09:49:44.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0949e0 con 0x7f21c4003cb0 2021-03-09T09:49:44.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0512f0 con 0x7f21cc042f20 2021-03-09T09:49:44.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:44.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:44.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:44.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:44.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.616+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:44.628 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23709 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:44.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:44.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:44.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:44.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:44.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d4f0 con 0x7f21cc042f20 2021-03-09T09:49:44.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094b50 con 0x7f21cc042f20 2021-03-09T09:49:44.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.620+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aa60 secure :-1 s=READY pgs=7035 cs=0 l=1 rev1=1 rx=0x7f21dc082010 tx=0x7f21dc019400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:44.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc057ea0 con 0x7f21cc042f20 2021-03-09T09:49:44.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21cc042f20 2021-03-09T09:49:44.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:49:44.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.628+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:44.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.632+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:44.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:44.632+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:49.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:49.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:49.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aa60 secure :-1 s=READY pgs=7035 cs=0 l=1 rev1=1 rx=0x7f21dc082010 tx=0x7f21dc019400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:49.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aa60 secure :-1 s=READY pgs=7035 cs=0 l=1 rev1=1 rx=0x7f21dc082010 tx=0x7f21dc019400).stop 2021-03-09T09:49:49.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01aa60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:49.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01aa60 unknown :-1 s=CLOSED pgs=7035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:49.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:49.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:49.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0941b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:49.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cf20 con 0x7f21cc005e30 2021-03-09T09:49:49.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027220 con 0x7f21c4003cb0 2021-03-09T09:49:49.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0946a0 con 0x7f21cc052490 2021-03-09T09:49:49.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:49.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:49.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:49.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:49.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0941b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:49.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:49.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:49.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:49.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:49.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094810 con 0x7f21cc052490 2021-03-09T09:49:49.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d4f0 con 0x7f21cc052490 2021-03-09T09:49:49.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0941b0 secure :-1 s=READY pgs=7046 cs=0 l=1 rev1=1 rx=0x7f21cc01af50 tx=0x7f21dc02aba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:49.642 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:49.624+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23743 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:49.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc052490 2021-03-09T09:49:49.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc052490 2021-03-09T09:49:49.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.632+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:49:49.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.632+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:49.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.636+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:49.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:49.636+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:52.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:52.455 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:49:54.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0941b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:54.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0941b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:54.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0941b0 secure :-1 s=READY pgs=7046 cs=0 l=1 rev1=1 rx=0x7f21cc01af50 tx=0x7f21dc02aba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:54.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0941b0 secure :-1 s=READY pgs=7046 cs=0 l=1 rev1=1 rx=0x7f21cc01af50 tx=0x7f21dc02aba0).stop 2021-03-09T09:49:54.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0941b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:54.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0941b0 unknown :-1 s=CLOSED pgs=7046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:54.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:54.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:54.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:54.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:49:54.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b570 con 0x7f21c4003cb0 2021-03-09T09:49:54.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b6b0 con 0x7f21cc042f20 2021-03-09T09:49:54.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:54.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:54.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:54.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:54.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:54.637 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:54.628+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23783 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:54.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:54.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:54.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:54.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:54.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc083d30 con 0x7f21cc042f20 2021-03-09T09:49:54.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094810 con 0x7f21cc042f20 2021-03-09T09:49:54.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7059 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc010d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:54.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.632+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc042f20 2021-03-09T09:49:54.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc042f20 2021-03-09T09:49:54.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052ca0 con 0x7f21cc042f20 2021-03-09T09:49:54.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.640+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:54.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.640+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:54.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:54.640+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:49:59.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:59.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:59.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7059 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc010d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:59.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7059 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc010d60).stop 2021-03-09T09:49:59.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:59.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:59.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:59.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:59.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:49:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b570 con 0x7f21c4003cb0 2021-03-09T09:49:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b6b0 con 0x7f21cc052490 2021-03-09T09:49:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:49:59.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:59.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:49:59.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:59.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.632+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:59.706 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23818 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:49:59.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:59.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:59.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094b50 con 0x7f21cc052490 2021-03-09T09:49:59.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc085f30 con 0x7f21cc052490 2021-03-09T09:49:59.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094110 secure :-1 s=READY pgs=7065 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4016fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:59.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408a220 con 0x7f21cc052490 2021-03-09T09:49:59.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408a380 con 0x7f21cc052490 2021-03-09T09:49:59.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408a220 con 0x7f21cc052490 2021-03-09T09:49:59.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.644+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:49:59.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.644+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:49:59.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:49:59.644+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:04.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:04.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:04.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094110 secure :-1 s=READY pgs=7065 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4016fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:04.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094110 secure :-1 s=READY pgs=7065 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4016fb0).stop 2021-03-09T09:50:04.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:04.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094110 unknown :-1 s=CLOSED pgs=7065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:04.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:04.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:04.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:04.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:50:04.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ec0 con 0x7f21c4003cb0 2021-03-09T09:50:04.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084000 con 0x7f21cc042f20 2021-03-09T09:50:04.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:04.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:04.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:04.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:04.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:04.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:04.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:04.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:04.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:04.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084170 con 0x7f21cc042f20 2021-03-09T09:50:04.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094b50 con 0x7f21cc042f20 2021-03-09T09:50:04.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.636+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7075 cs=0 l=1 rev1=1 rx=0x7f21e407f210 tx=0x7f21e4009230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:04.648 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:04.640+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23857 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:04.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21cc042f20 2021-03-09T09:50:04.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409cab0 con 0x7f21cc042f20 2021-03-09T09:50:04.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cab0 con 0x7f21cc042f20 2021-03-09T09:50:04.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.644+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:04.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.644+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:04.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:04.644+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:09.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:09.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:09.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7075 cs=0 l=1 rev1=1 rx=0x7f21e407f210 tx=0x7f21e4009230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:09.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7075 cs=0 l=1 rev1=1 rx=0x7f21e407f210 tx=0x7f21e4009230).stop 2021-03-09T09:50:09.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:09.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:09.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:09.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:09.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:09.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21cc005e30 2021-03-09T09:50:09.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009890 con 0x7f21c4003cb0 2021-03-09T09:50:09.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0099d0 con 0x7f21cc052490 2021-03-09T09:50:09.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:09.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:09.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:09.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:09.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.640+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:09.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:09.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:09.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:09.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:09.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084300 con 0x7f21cc052490 2021-03-09T09:50:09.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc083d30 con 0x7f21cc052490 2021-03-09T09:50:09.652 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23892 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:09.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094090 secure :-1 s=READY pgs=7088 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc062fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:09.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.648+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02ac10 con 0x7f21cc052490 2021-03-09T09:50:09.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.648+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc072dc0 con 0x7f21cc052490 2021-03-09T09:50:09.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.648+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:50:09.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.648+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:09.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.652+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:09.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:09.652+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:14.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:14.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:14.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094090 secure :-1 s=READY pgs=7088 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc062fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:14.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094090 secure :-1 s=READY pgs=7088 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc062fc0).stop 2021-03-09T09:50:14.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc094090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:14.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc094090 unknown :-1 s=CLOSED pgs=7088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:14.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:14.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:14.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:14.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec30 con 0x7f21cc005e30 2021-03-09T09:50:14.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d4f0 con 0x7f21c4003cb0 2021-03-09T09:50:14.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc042f20 2021-03-09T09:50:14.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:14.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:14.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:14.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:14.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.644+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:14.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:14.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:14.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:14.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:14.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009860 con 0x7f21cc042f20 2021-03-09T09:50:14.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084170 con 0x7f21cc042f20 2021-03-09T09:50:14.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7097 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4091200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:14.656 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23928 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:14.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.648+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d130 con 0x7f21cc042f20 2021-03-09T09:50:14.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d130 con 0x7f21cc042f20 2021-03-09T09:50:14.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028280 con 0x7f21cc042f20 2021-03-09T09:50:14.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.656+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:14.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.656+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:14.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:14.656+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:19.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:19.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:19.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7097 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4091200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:19.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7097 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e4091200).stop 2021-03-09T09:50:19.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:19.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:19.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:19.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:19.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:19.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027140 con 0x7f21cc005e30 2021-03-09T09:50:19.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027280 con 0x7f21c4003cb0 2021-03-09T09:50:19.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094390 con 0x7f21cc052490 2021-03-09T09:50:19.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:19.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:19.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:19.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:19.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:19.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:19.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:19.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:19.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:19.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094500 con 0x7f21cc052490 2021-03-09T09:50:19.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009860 con 0x7f21cc052490 2021-03-09T09:50:19.661 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23966 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:19.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093ea0 secure :-1 s=READY pgs=7108 cs=0 l=1 rev1=1 rx=0x7f21e40b0290 tx=0x7f21e4057e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:19.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016dd0 con 0x7f21cc052490 2021-03-09T09:50:19.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21cc052490 2021-03-09T09:50:19.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.660+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21cc052490 2021-03-09T09:50:19.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.660+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:19.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.660+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:19.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:19.660+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:22.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:22.494 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:24.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:24.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:24.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093ea0 secure :-1 s=READY pgs=7108 cs=0 l=1 rev1=1 rx=0x7f21e40b0290 tx=0x7f21e4057e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:24.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093ea0 secure :-1 s=READY pgs=7108 cs=0 l=1 rev1=1 rx=0x7f21e40b0290 tx=0x7f21e4057e80).stop 2021-03-09T09:50:24.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:24.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093ea0 unknown :-1 s=CLOSED pgs=7108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:24.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:24.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:24.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:24.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052350 con 0x7f21cc005e30 2021-03-09T09:50:24.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21c4003cb0 2021-03-09T09:50:24.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094480 con 0x7f21cc042f20 2021-03-09T09:50:24.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:24.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:24.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:24.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:24.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:24.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:24.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:24.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:24.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:24.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026fa0 con 0x7f21cc042f20 2021-03-09T09:50:24.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009860 con 0x7f21cc042f20 2021-03-09T09:50:24.666 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24002 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:24.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7120 cs=0 l=1 rev1=1 rx=0x7f21e4017160 tx=0x7f21e4005f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:24.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.660+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40555f0 con 0x7f21cc042f20 2021-03-09T09:50:24.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40555f0 con 0x7f21cc042f20 2021-03-09T09:50:24.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.668+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T09:50:24.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.668+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:24.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.668+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:24.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:24.668+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:29.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:29.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:29.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7120 cs=0 l=1 rev1=1 rx=0x7f21e4017160 tx=0x7f21e4005f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:29.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7120 cs=0 l=1 rev1=1 rx=0x7f21e4017160 tx=0x7f21e4005f30).stop 2021-03-09T09:50:29.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:29.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:29.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:29.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:29.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:29.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6d0 con 0x7f21cc005e30 2021-03-09T09:50:29.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043300 con 0x7f21c4003cb0 2021-03-09T09:50:29.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009860 con 0x7f21cc052490 2021-03-09T09:50:29.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:29.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:29.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:29.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:29.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:29.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:29.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:29.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:29.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:29.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044920 con 0x7f21cc052490 2021-03-09T09:50:29.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084300 con 0x7f21cc052490 2021-03-09T09:50:29.669 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24040 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:29.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.659+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093e00 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 rx=0x7f21cc042b00 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:29.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085130 con 0x7f21cc052490 2021-03-09T09:50:29.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc072a50 con 0x7f21cc052490 2021-03-09T09:50:29.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.667+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc052490 2021-03-09T09:50:29.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.667+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:29.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.667+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:29.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:29.667+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:34.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:34.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:34.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093e00 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 rx=0x7f21cc042b00 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:34.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093e00 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 rx=0x7f21cc042b00 tx=0x7f21dc08a120).stop 2021-03-09T09:50:34.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:34.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093e00 unknown :-1 s=CLOSED pgs=7128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:34.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:34.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:34.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.663+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:34.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f270 con 0x7f21cc005e30 2021-03-09T09:50:34.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009860 con 0x7f21c4003cb0 2021-03-09T09:50:34.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03efd0 con 0x7f21cc042f20 2021-03-09T09:50:34.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:34.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:34.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:34.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:34.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:34.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:34.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:34.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:34.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:34.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f140 con 0x7f21cc042f20 2021-03-09T09:50:34.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026fa0 con 0x7f21cc042f20 2021-03-09T09:50:34.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.667+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7140 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40085d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:34.675 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:34.671+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24076 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:34.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21cc042f20 2021-03-09T09:50:34.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40555f0 con 0x7f21cc042f20 2021-03-09T09:50:34.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b0180 con 0x7f21cc042f20 2021-03-09T09:50:34.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.679+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:34.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.679+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:34.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:34.679+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:39.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7140 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40085d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7140 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40085d0).stop 2021-03-09T09:50:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:39.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:39.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:39.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:39.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:39.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a730 con 0x7f21cc005e30 2021-03-09T09:50:39.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f270 con 0x7f21c4003cb0 2021-03-09T09:50:39.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009860 con 0x7f21cc052490 2021-03-09T09:50:39.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:39.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:39.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:39.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:39.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:39.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:39.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:39.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:39.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:39.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03ef80 con 0x7f21cc052490 2021-03-09T09:50:39.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044920 con 0x7f21cc052490 2021-03-09T09:50:39.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.671+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093c10 secure :-1 s=READY pgs=7150 cs=0 l=1 rev1=1 rx=0x7f21cc01b8b0 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:39.679 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:39.675+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24114 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:39.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc052490 2021-03-09T09:50:39.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06b670 con 0x7f21cc052490 2021-03-09T09:50:39.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc052490 2021-03-09T09:50:39.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.679+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:39.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.679+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:39.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:39.679+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:44.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:44.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093c10 secure :-1 s=READY pgs=7150 cs=0 l=1 rev1=1 rx=0x7f21cc01b8b0 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093c10 secure :-1 s=READY pgs=7150 cs=0 l=1 rev1=1 rx=0x7f21cc01b8b0 tx=0x7f21dc07ff90).stop 2021-03-09T09:50:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093c10 unknown :-1 s=CLOSED pgs=7150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f270 con 0x7f21cc005e30 2021-03-09T09:50:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044780 con 0x7f21c4003cb0 2021-03-09T09:50:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0448c0 con 0x7f21cc042f20 2021-03-09T09:50:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:44.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:44.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:44.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:44.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:44.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:44.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:44.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:44.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:44.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044a00 con 0x7f21cc042f20 2021-03-09T09:50:44.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.675+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03ef80 con 0x7f21cc042f20 2021-03-09T09:50:44.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.679+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7160 cs=0 l=1 rev1=1 rx=0x7f21cc04ce50 tx=0x7f21dc085ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:44.684 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:44.679+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24151 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:44.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21cc042f20 2021-03-09T09:50:44.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0212e0 con 0x7f21cc042f20 2021-03-09T09:50:44.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.687+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21cc042f20 2021-03-09T09:50:44.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.687+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:44.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.687+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:44.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:44.687+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:49.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:49.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:49.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7160 cs=0 l=1 rev1=1 rx=0x7f21cc04ce50 tx=0x7f21dc085ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:49.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7160 cs=0 l=1 rev1=1 rx=0x7f21cc04ce50 tx=0x7f21dc085ef0).stop 2021-03-09T09:50:49.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:49.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:49.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:49.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:49.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:49.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f270 con 0x7f21cc005e30 2021-03-09T09:50:49.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094060 con 0x7f21c4003cb0 2021-03-09T09:50:49.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013230 con 0x7f21cc052490 2021-03-09T09:50:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.679+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013370 con 0x7f21cc052490 2021-03-09T09:50:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044a00 con 0x7f21cc052490 2021-03-09T09:50:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093b70 secure :-1 s=READY pgs=7171 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc056350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:49.690 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24188 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:49.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc052490 2021-03-09T09:50:49.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21cc052490 2021-03-09T09:50:49.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc052490 2021-03-09T09:50:49.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.691+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:49.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.691+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:49.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:49.691+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:52.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:52.539 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:50:54.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:54.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:54.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093b70 secure :-1 s=READY pgs=7171 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc056350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:54.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093b70 secure :-1 s=READY pgs=7171 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc056350).stop 2021-03-09T09:50:54.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc093b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:54.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc093b70 unknown :-1 s=CLOSED pgs=7171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:54.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:54.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:54.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:54.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013230 con 0x7f21cc005e30 2021-03-09T09:50:54.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042d20 con 0x7f21c4003cb0 2021-03-09T09:50:54.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01f270 con 0x7f21cc042f20 2021-03-09T09:50:54.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:54.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:54.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:54.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:54.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:54.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:54.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:54.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:54.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:54.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044a00 con 0x7f21cc042f20 2021-03-09T09:50:54.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013370 con 0x7f21cc042f20 2021-03-09T09:50:54.694 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:54.687+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24223 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:54.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7186 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0561d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:54.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:50:54.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc096000 con 0x7f21cc042f20 2021-03-09T09:50:54.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:50:54.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.699+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:54.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.699+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:54.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:54.699+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:50:59.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:59.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:59.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7186 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0561d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:59.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 secure :-1 s=READY pgs=7186 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0561d0).stop 2021-03-09T09:50:59.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01a9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:59.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01a9f0 unknown :-1 s=CLOSED pgs=7186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:59.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:59.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:59.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:59.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d1d0 con 0x7f21cc005e30 2021-03-09T09:50:59.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9b0 con 0x7f21c4003cb0 2021-03-09T09:50:59.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042d00 con 0x7f21cc052490 2021-03-09T09:50:59.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:50:59.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:50:59.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:59.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:59.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:59.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:59.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:59.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:59.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:59.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f270 con 0x7f21cc052490 2021-03-09T09:50:59.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044a00 con 0x7f21cc052490 2021-03-09T09:50:59.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7192 cs=0 l=1 rev1=1 rx=0x7f21dc056510 tx=0x7f21dc0ace50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:59.700 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:50:59.691+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24262 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:50:59.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.695+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21cc052490 2021-03-09T09:50:59.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.695+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc052490 2021-03-09T09:50:59.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc052490 2021-03-09T09:50:59.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.699+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:50:59.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.699+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:50:59.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:50:59.699+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:04.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:04.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:04.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7192 cs=0 l=1 rev1=1 rx=0x7f21dc056510 tx=0x7f21dc0ace50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:04.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7192 cs=0 l=1 rev1=1 rx=0x7f21dc056510 tx=0x7f21dc0ace50).stop 2021-03-09T09:51:04.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:04.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=CLOSED pgs=7192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:04.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:04.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:04.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:04.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d1d0 con 0x7f21cc005e30 2021-03-09T09:51:04.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0307d0 con 0x7f21c4003cb0 2021-03-09T09:51:04.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9b0 con 0x7f21cc042f20 2021-03-09T09:51:04.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:04.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:04.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:04.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:04.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:04.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:04.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:04.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:04.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:04.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044a00 con 0x7f21cc042f20 2021-03-09T09:51:04.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f270 con 0x7f21cc042f20 2021-03-09T09:51:04.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.695+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035600 secure :-1 s=READY pgs=7203 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:04.703 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:04.699+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24298 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:04.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc042f20 2021-03-09T09:51:04.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032250 con 0x7f21cc042f20 2021-03-09T09:51:04.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc042f20 2021-03-09T09:51:04.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.703+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:04.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.703+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:04.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:04.703+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:09.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:51:09.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:09.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035600 secure :-1 s=READY pgs=7203 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:09.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035600 secure :-1 s=READY pgs=7203 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc077de0).stop 2021-03-09T09:51:09.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:09.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035600 unknown :-1 s=CLOSED pgs=7203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:09.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:09.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:09.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:09.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e9b0 con 0x7f21cc005e30 2021-03-09T09:51:09.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027e80 con 0x7f21c4003cb0 2021-03-09T09:51:09.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ce0 con 0x7f21cc052490 2021-03-09T09:51:09.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:09.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:09.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:09.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:09.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.699+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:09.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:09.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:09.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:09.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:09.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f270 con 0x7f21cc052490 2021-03-09T09:51:09.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f140 con 0x7f21cc052490 2021-03-09T09:51:09.707 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24336 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:09.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7216 cs=0 l=1 rev1=1 rx=0x7f21cc058c10 tx=0x7f21e4016b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:09.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:51:09.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:51:09.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc052490 2021-03-09T09:51:09.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.711+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:09.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.711+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:09.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:09.711+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:14.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:14.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:14.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7216 cs=0 l=1 rev1=1 rx=0x7f21cc058c10 tx=0x7f21e4016b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:14.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7216 cs=0 l=1 rev1=1 rx=0x7f21cc058c10 tx=0x7f21e4016b20).stop 2021-03-09T09:51:14.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:14.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=CLOSED pgs=7216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:14.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:14.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:14.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:14.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ce0 con 0x7f21cc005e30 2021-03-09T09:51:14.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f6d0 con 0x7f21c4003cb0 2021-03-09T09:51:14.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f640 con 0x7f21cc042f20 2021-03-09T09:51:14.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:14.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:14.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:14.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:14.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.703+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:14.710 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24371 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:14.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:14.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:14.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:14.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:14.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f780 con 0x7f21cc042f20 2021-03-09T09:51:14.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044a00 con 0x7f21cc042f20 2021-03-09T09:51:14.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.707+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035400 secure :-1 s=READY pgs=7229 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:14.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.711+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:51:14.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032250 con 0x7f21cc042f20 2021-03-09T09:51:14.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.715+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:51:14.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.715+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:14.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.715+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:14.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:14.715+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:19.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:19.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:19.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035400 secure :-1 s=READY pgs=7229 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:19.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035400 secure :-1 s=READY pgs=7229 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc08a120).stop 2021-03-09T09:51:19.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:19.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035400 unknown :-1 s=CLOSED pgs=7229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:19.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:19.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:19.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:19.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ce0 con 0x7f21cc005e30 2021-03-09T09:51:19.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f6d0 con 0x7f21c4003cb0 2021-03-09T09:51:19.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035de0 con 0x7f21cc052490 2021-03-09T09:51:19.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:19.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:19.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:19.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:19.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:19.715 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24410 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:19.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:19.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:19.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:19.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:19.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044a00 con 0x7f21cc052490 2021-03-09T09:51:19.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f780 con 0x7f21cc052490 2021-03-09T09:51:19.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0358f0 secure :-1 s=READY pgs=7237 cs=0 l=1 rev1=1 rx=0x7f21dc021570 tx=0x7f21dc021420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:19.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:51:19.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc052490 2021-03-09T09:51:19.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc052490 2021-03-09T09:51:19.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.718+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:19.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.722+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:19.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:19.722+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:22.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:22.593 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:24.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:24.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:24.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0358f0 secure :-1 s=READY pgs=7237 cs=0 l=1 rev1=1 rx=0x7f21dc021570 tx=0x7f21dc021420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:24.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0358f0 secure :-1 s=READY pgs=7237 cs=0 l=1 rev1=1 rx=0x7f21dc021570 tx=0x7f21dc021420).stop 2021-03-09T09:51:24.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc0358f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:24.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc0358f0 unknown :-1 s=CLOSED pgs=7237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:24.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:24.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:24.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:24.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dab0 con 0x7f21cc005e30 2021-03-09T09:51:24.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00da20 con 0x7f21c4003cb0 2021-03-09T09:51:24.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ce0 con 0x7f21cc042f20 2021-03-09T09:51:24.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:24.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.710+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:24.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:24.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:24.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:24.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:24.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:24.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:24.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:24.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f5f0 con 0x7f21cc042f20 2021-03-09T09:51:24.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044a00 con 0x7f21cc042f20 2021-03-09T09:51:24.721 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24445 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:24.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035360 secure :-1 s=READY pgs=7247 cs=0 l=1 rev1=1 rx=0x7f21dc02f510 tx=0x7f21dc05bc80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:24.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:51:24.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc054000 con 0x7f21cc042f20 2021-03-09T09:51:24.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:51:24.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.722+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:24.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.722+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:24.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:24.722+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:29.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:29.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:29.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035360 secure :-1 s=READY pgs=7247 cs=0 l=1 rev1=1 rx=0x7f21dc02f510 tx=0x7f21dc05bc80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:29.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035360 secure :-1 s=READY pgs=7247 cs=0 l=1 rev1=1 rx=0x7f21dc02f510 tx=0x7f21dc05bc80).stop 2021-03-09T09:51:29.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc035360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:29.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc035360 unknown :-1 s=CLOSED pgs=7247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:29.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:29.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:29.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:29.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:51:29.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02cce0 con 0x7f21c4003cb0 2021-03-09T09:51:29.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ce0 con 0x7f21cc052490 2021-03-09T09:51:29.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:29.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:29.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:29.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:29.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:29.723 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24484 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:29.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:29.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:29.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:29.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:29.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044a00 con 0x7f21cc052490 2021-03-09T09:51:29.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f270 con 0x7f21cc052490 2021-03-09T09:51:29.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7261 cs=0 l=1 rev1=1 rx=0x7f21e4009c40 tx=0x7f21e40a51d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:29.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007160 con 0x7f21cc052490 2021-03-09T09:51:29.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007160 con 0x7f21cc052490 2021-03-09T09:51:29.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc052490 2021-03-09T09:51:29.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.726+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:29.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:29.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:29.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:34.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:51:34.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:34.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7261 cs=0 l=1 rev1=1 rx=0x7f21e4009c40 tx=0x7f21e40a51d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:34.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7261 cs=0 l=1 rev1=1 rx=0x7f21e4009c40 tx=0x7f21e40a51d0).stop 2021-03-09T09:51:34.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:34.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=CLOSED pgs=7261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:34.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:34.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:34.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:34.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02cce0 con 0x7f21cc005e30 2021-03-09T09:51:34.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044d90 con 0x7f21c4003cb0 2021-03-09T09:51:34.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ce0 con 0x7f21cc042f20 2021-03-09T09:51:34.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:34.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:34.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:34.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.718+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:34.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:34.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:34.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:34.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:34.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:34.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04dc00 con 0x7f21cc042f20 2021-03-09T09:51:34.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f5f0 con 0x7f21cc042f20 2021-03-09T09:51:34.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7267 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:34.730 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:34.722+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24519 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:34.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02b230 con 0x7f21cc042f20 2021-03-09T09:51:34.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056210 con 0x7f21cc042f20 2021-03-09T09:51:34.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc042f20 2021-03-09T09:51:34.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.730+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:34.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:34.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:34.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:39.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:39.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:39.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7267 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:39.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7267 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077de0).stop 2021-03-09T09:51:39.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:39.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=CLOSED pgs=7267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:39.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:39.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:39.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:39.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:39.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:39.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044d90 con 0x7f21cc005e30 2021-03-09T09:51:39.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21c4003cb0 2021-03-09T09:51:39.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02cce0 con 0x7f21cc052490 2021-03-09T09:51:39.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:39.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:39.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:39.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:39.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:39.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:39.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:39.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f270 con 0x7f21cc052490 2021-03-09T09:51:39.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04dc00 con 0x7f21cc052490 2021-03-09T09:51:39.734 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:39.726+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24558 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:39.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7278 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc0528a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:39.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3c50 con 0x7f21cc052490 2021-03-09T09:51:39.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3c50 con 0x7f21cc052490 2021-03-09T09:51:39.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004820 con 0x7f21cc052490 2021-03-09T09:51:39.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.734+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:39.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.734+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:39.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:39.734+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:44.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:44.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:44.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7278 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc0528a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:44.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 secure :-1 s=READY pgs=7278 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc0528a0).stop 2021-03-09T09:51:44.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 msgr2=0x7f21cc013230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:44.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc052490 0x7f21cc013230 unknown :-1 s=CLOSED pgs=7278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:44.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:44.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:44.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:44.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044d90 con 0x7f21cc005e30 2021-03-09T09:51:44.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21c4003cb0 2021-03-09T09:51:44.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02cce0 con 0x7f21cc042f20 2021-03-09T09:51:44.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:44.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:44.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:44.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:44.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:44.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:44.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:44.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:44.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:44.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:51:44.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044a00 con 0x7f21cc042f20 2021-03-09T09:51:44.739 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:44.730+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24593 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:44.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7288 cs=0 l=1 rev1=1 rx=0x7f21cc021460 tx=0x7f21e40314b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:44.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T09:51:44.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T09:51:44.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T09:51:44.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.742+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:44.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.742+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:44.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:44.742+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:49.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:49.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:49.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7288 cs=0 l=1 rev1=1 rx=0x7f21cc021460 tx=0x7f21e40314b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:49.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7288 cs=0 l=1 rev1=1 rx=0x7f21cc021460 tx=0x7f21e40314b0).stop 2021-03-09T09:51:49.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:49.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=CLOSED pgs=7288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:49.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:49.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:49.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:49.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044d90 con 0x7f21cc005e30 2021-03-09T09:51:49.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:49.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21c4003cb0 2021-03-09T09:51:49.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02cce0 con 0x7f21cc02f1a0 2021-03-09T09:51:49.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:49.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:49.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:49.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:49.741 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:49.734+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24632 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:49.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:49.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:49.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:49.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:49.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013590 con 0x7f21cc02f1a0 2021-03-09T09:51:49.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f270 con 0x7f21cc02f1a0 2021-03-09T09:51:49.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013c10 secure :-1 s=READY pgs=7304 cs=0 l=1 rev1=1 rx=0x7f21dc06a850 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:49.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc02f1a0 2021-03-09T09:51:49.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6350 con 0x7f21cc02f1a0 2021-03-09T09:51:49.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:51:49.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.746+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:49.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:49.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:49.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:52.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:52.638 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:51:54.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:54.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:54.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013c10 secure :-1 s=READY pgs=7304 cs=0 l=1 rev1=1 rx=0x7f21dc06a850 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:54.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013c10 secure :-1 s=READY pgs=7304 cs=0 l=1 rev1=1 rx=0x7f21dc06a850 tx=0x7f21dc02df50).stop 2021-03-09T09:51:54.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:54.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013c10 unknown :-1 s=CLOSED pgs=7304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:54.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:54.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:54.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:54.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc005e30 2021-03-09T09:51:54.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044d90 con 0x7f21c4003cb0 2021-03-09T09:51:54.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:51:54.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:54.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:54.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:54.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:54.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:54.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:54.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:54.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:54.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:54.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01f270 con 0x7f21cc042f20 2021-03-09T09:51:54.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013590 con 0x7f21cc042f20 2021-03-09T09:51:54.747 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24667 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:54.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7311 cs=0 l=1 rev1=1 rx=0x7f21dc0b62e0 tx=0x7f21dc0b6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:54.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc042f20 2021-03-09T09:51:54.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T09:51:54.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:51:54.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.750+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:54.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.750+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:54.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:54.750+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:51:59.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:59.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:59.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7311 cs=0 l=1 rev1=1 rx=0x7f21dc0b62e0 tx=0x7f21dc0b6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:59.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 secure :-1 s=READY pgs=7311 cs=0 l=1 rev1=1 rx=0x7f21dc0b62e0 tx=0x7f21dc0b6190).stop 2021-03-09T09:51:59.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc013720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:59.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc013720 unknown :-1 s=CLOSED pgs=7311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:59.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:59.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc052460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:59.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:59.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc005e30 2021-03-09T09:51:59.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009b40 con 0x7f21c4003cb0 2021-03-09T09:51:59.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21cc02f1a0 2021-03-09T09:51:59.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:51:59.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:59.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:51:59.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:59.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc052460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:59.749 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24706 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:51:59.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:59.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:59.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:59.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:59.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02cc90 con 0x7f21cc02f1a0 2021-03-09T09:51:59.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:51:59.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc052460 secure :-1 s=READY pgs=7324 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4017240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:59.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc02f1a0 2021-03-09T09:51:59.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21cc02f1a0 2021-03-09T09:51:59.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc02f1a0 2021-03-09T09:51:59.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.754+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:51:59.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:51:59.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:51:59.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:04.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc052460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:52:04.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc052460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:04.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc052460 secure :-1 s=READY pgs=7324 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4017240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:04.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc052460 secure :-1 s=READY pgs=7324 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4017240).stop 2021-03-09T09:52:04.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc052460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:04.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc052460 unknown :-1 s=CLOSED pgs=7324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:04.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:04.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:04.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:04.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc005e30 2021-03-09T09:52:04.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009b40 con 0x7f21c4003cb0 2021-03-09T09:52:04.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21cc042f20 2021-03-09T09:52:04.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:04.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:04.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:04.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:04.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.746+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:04.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:04.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:04.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:04.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:04.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:52:04.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01f270 con 0x7f21cc042f20 2021-03-09T09:52:04.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 secure :-1 s=READY pgs=7332 cs=0 l=1 rev1=1 rx=0x7f21cc02ead0 tx=0x7f21dc069af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:04.756 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24741 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:04.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:52:04.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b82c0 con 0x7f21cc042f20 2021-03-09T09:52:04.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc042f20 2021-03-09T09:52:04.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.758+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:04.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:04.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:04.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:09.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:09.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:09.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 secure :-1 s=READY pgs=7332 cs=0 l=1 rev1=1 rx=0x7f21cc02ead0 tx=0x7f21dc069af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:09.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 secure :-1 s=READY pgs=7332 cs=0 l=1 rev1=1 rx=0x7f21cc02ead0 tx=0x7f21dc069af0).stop 2021-03-09T09:52:09.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01df70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:09.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 unknown :-1 s=CLOSED pgs=7332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:09.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:09.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc056c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:09.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:09.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc005e30 2021-03-09T09:52:09.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009b40 con 0x7f21c4003cb0 2021-03-09T09:52:09.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21cc02f1a0 2021-03-09T09:52:09.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:09.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:09.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc056c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:09.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:09.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:09.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:09.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:09.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:09.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:09.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086310 con 0x7f21cc02f1a0 2021-03-09T09:52:09.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02cc90 con 0x7f21cc02f1a0 2021-03-09T09:52:09.761 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24780 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:09.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.754+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc056c00 secure :-1 s=READY pgs=7343 cs=0 l=1 rev1=1 rx=0x7f21cc04f980 tx=0x7f21e4020f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:09.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009ae0 con 0x7f21cc02f1a0 2021-03-09T09:52:09.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009ae0 con 0x7f21cc02f1a0 2021-03-09T09:52:09.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T09:52:09.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.762+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:09.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:09.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:09.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:14.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc056c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:14.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc056c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:14.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc056c00 secure :-1 s=READY pgs=7343 cs=0 l=1 rev1=1 rx=0x7f21cc04f980 tx=0x7f21e4020f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:14.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc056c00 secure :-1 s=READY pgs=7343 cs=0 l=1 rev1=1 rx=0x7f21cc04f980 tx=0x7f21e4020f60).stop 2021-03-09T09:52:14.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc056c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:14.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc056c00 unknown :-1 s=CLOSED pgs=7343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:14.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:14.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:14.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:14.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc005e30 2021-03-09T09:52:14.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053170 con 0x7f21c4003cb0 2021-03-09T09:52:14.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21cc042f20 2021-03-09T09:52:14.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:14.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:14.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:14.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:14.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:14.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:14.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:14.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:14.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:14.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:52:14.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:52:14.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 secure :-1 s=READY pgs=7353 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:14.766 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:14.757+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24815 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:14.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099020 con 0x7f21cc042f20 2021-03-09T09:52:14.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ab540 con 0x7f21cc042f20 2021-03-09T09:52:14.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc042f20 2021-03-09T09:52:14.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.765+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:14.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.765+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:14.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:14.765+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:19.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:19.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:19.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 secure :-1 s=READY pgs=7353 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:19.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 secure :-1 s=READY pgs=7353 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc080000).stop 2021-03-09T09:52:19.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01df70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:19.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01df70 unknown :-1 s=CLOSED pgs=7353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:19.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:19.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:19.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:19.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc005e30 2021-03-09T09:52:19.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21c4003cb0 2021-03-09T09:52:19.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00eca0 con 0x7f21cc02f1a0 2021-03-09T09:52:19.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:19.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:19.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:19.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:19.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:19.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:19.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:19.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:19.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:19.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:52:19.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02cc90 con 0x7f21cc02f1a0 2021-03-09T09:52:19.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013720 secure :-1 s=READY pgs=7364 cs=0 l=1 rev1=1 rx=0x7f21cc035a50 tx=0x7f21e40a63f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:19.770 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:19.761+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24854 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:19.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21cc02f1a0 2021-03-09T09:52:19.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4086db0 con 0x7f21cc02f1a0 2021-03-09T09:52:19.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc02f1a0 2021-03-09T09:52:19.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.769+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:19.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.773+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:19.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:19.773+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:22.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:22.686 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:24.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:24.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:24.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013720 secure :-1 s=READY pgs=7364 cs=0 l=1 rev1=1 rx=0x7f21cc035a50 tx=0x7f21e40a63f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:24.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013720 secure :-1 s=READY pgs=7364 cs=0 l=1 rev1=1 rx=0x7f21cc035a50 tx=0x7f21e40a63f0).stop 2021-03-09T09:52:24.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:24.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013720 unknown :-1 s=CLOSED pgs=7364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:24.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:24.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:24.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:24.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cda0 con 0x7f21cc005e30 2021-03-09T09:52:24.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21c4003cb0 2021-03-09T09:52:24.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f0f0 con 0x7f21cc042f20 2021-03-09T09:52:24.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:24.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:24.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:24.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:24.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:24.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:24.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:24.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:24.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:24.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f970 con 0x7f21cc042f20 2021-03-09T09:52:24.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:52:24.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056bb0 secure :-1 s=READY pgs=7374 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:24.775 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:24.769+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24890 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:24.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004c30 con 0x7f21cc042f20 2021-03-09T09:52:24.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3350 con 0x7f21cc042f20 2021-03-09T09:52:24.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:52:24.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.773+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:24.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:24.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:24.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:29.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc056bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:29.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc056bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:29.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056bb0 secure :-1 s=READY pgs=7374 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:29.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056bb0 secure :-1 s=READY pgs=7374 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010e10).stop 2021-03-09T09:52:29.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc056bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:29.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056bb0 unknown :-1 s=CLOSED pgs=7374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:29.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:29.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:29.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:29.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cda0 con 0x7f21cc005e30 2021-03-09T09:52:29.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fe80 con 0x7f21c4003cb0 2021-03-09T09:52:29.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089550 con 0x7f21cc02f1a0 2021-03-09T09:52:29.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:29.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:29.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:29.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:29.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:29.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:29.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:29.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:29.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:29.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc02f1a0 2021-03-09T09:52:29.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:52:29.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7384 cs=0 l=1 rev1=1 rx=0x7f21e4043050 tx=0x7f21e4092420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:29.779 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:29.773+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24928 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:29.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5510 con 0x7f21cc02f1a0 2021-03-09T09:52:29.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5510 con 0x7f21cc02f1a0 2021-03-09T09:52:29.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4005500 con 0x7f21cc02f1a0 2021-03-09T09:52:29.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.777+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:29.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:29.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:29.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:34.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:52:34.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:34.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7384 cs=0 l=1 rev1=1 rx=0x7f21e4043050 tx=0x7f21e4092420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:34.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7384 cs=0 l=1 rev1=1 rx=0x7f21e4043050 tx=0x7f21e4092420).stop 2021-03-09T09:52:34.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:34.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=CLOSED pgs=7384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cda0 con 0x7f21cc005e30 2021-03-09T09:52:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024330 con 0x7f21c4003cb0 2021-03-09T09:52:34.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ca50 con 0x7f21cc042f20 2021-03-09T09:52:34.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:34.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:34.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:34.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:34.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:34.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:34.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:34.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:34.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:34.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:52:34.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:52:34.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0569c0 secure :-1 s=READY pgs=7395 cs=0 l=1 rev1=1 rx=0x7f21cc009da0 tx=0x7f21e402b2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:34.784 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:34.777+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24964 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:34.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T09:52:34.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016e60 con 0x7f21cc042f20 2021-03-09T09:52:34.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T09:52:34.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.785+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:34.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.785+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:34.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:34.785+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:39.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:39.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:39.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0569c0 secure :-1 s=READY pgs=7395 cs=0 l=1 rev1=1 rx=0x7f21cc009da0 tx=0x7f21e402b2f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:39.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0569c0 secure :-1 s=READY pgs=7395 cs=0 l=1 rev1=1 rx=0x7f21cc009da0 tx=0x7f21e402b2f0).stop 2021-03-09T09:52:39.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0569c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:39.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0569c0 unknown :-1 s=CLOSED pgs=7395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:39.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:39.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:39.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:39.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cda0 con 0x7f21cc005e30 2021-03-09T09:52:39.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:39.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024330 con 0x7f21c4003cb0 2021-03-09T09:52:39.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028450 con 0x7f21cc02f1a0 2021-03-09T09:52:39.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:39.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:39.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:39.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:39.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:39.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:39.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:39.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:39.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089500 con 0x7f21cc02f1a0 2021-03-09T09:52:39.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f970 con 0x7f21cc02f1a0 2021-03-09T09:52:39.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7407 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:39.789 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25002 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:39.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc02f1a0 2021-03-09T09:52:39.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc02f1a0 2021-03-09T09:52:39.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc02f1a0 2021-03-09T09:52:39.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.789+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:39.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.789+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:39.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:39.789+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:44.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:44.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:44.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7407 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:44.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7407 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6120).stop 2021-03-09T09:52:44.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:44.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=CLOSED pgs=7407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:44.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:44.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:44.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:44.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e570 con 0x7f21cc005e30 2021-03-09T09:52:44.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057ac0 con 0x7f21c4003cb0 2021-03-09T09:52:44.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e0d0 con 0x7f21cc042f20 2021-03-09T09:52:44.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:44.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:44.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:44.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:44.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:44.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:44.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:44.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:44.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:44.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:52:44.794 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25037 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:44.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:52:44.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056680 secure :-1 s=READY pgs=7420 cs=0 l=1 rev1=1 rx=0x7f21e40a1ff0 tx=0x7f21e407ef60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:44.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401aee0 con 0x7f21cc042f20 2021-03-09T09:52:44.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401aee0 con 0x7f21cc042f20 2021-03-09T09:52:44.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21cc042f20 2021-03-09T09:52:44.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.797+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:44.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.797+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:44.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:44.797+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:49.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc056680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:49.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc056680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:49.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056680 secure :-1 s=READY pgs=7420 cs=0 l=1 rev1=1 rx=0x7f21e40a1ff0 tx=0x7f21e407ef60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:49.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056680 secure :-1 s=READY pgs=7420 cs=0 l=1 rev1=1 rx=0x7f21e40a1ff0 tx=0x7f21e407ef60).stop 2021-03-09T09:52:49.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc056680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:49.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc056680 unknown :-1 s=CLOSED pgs=7420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:49.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:49.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:49.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:49.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d500 con 0x7f21cc005e30 2021-03-09T09:52:49.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f7c0 con 0x7f21c4003cb0 2021-03-09T09:52:49.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21cc02f1a0 2021-03-09T09:52:49.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:49.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:49.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:49.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:49.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:49.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:49.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:49.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:52:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089500 con 0x7f21cc02f1a0 2021-03-09T09:52:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 rx=0x7f21dc057bf0 tx=0x7f21dc057c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:49.799 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:49.793+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25076 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:49.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc02f1a0 2021-03-09T09:52:49.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06d910 con 0x7f21cc02f1a0 2021-03-09T09:52:49.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:52:49.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.797+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:49.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.797+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:49.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:49.797+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:52.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:52.733 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:52:54.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:54.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:54.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 rx=0x7f21dc057bf0 tx=0x7f21dc057c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:54.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 rx=0x7f21dc057bf0 tx=0x7f21dc057c20).stop 2021-03-09T09:52:54.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc086110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:54.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc086110 unknown :-1 s=CLOSED pgs=7427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:54.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:54.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:54.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:54.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21cc005e30 2021-03-09T09:52:54.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089500 con 0x7f21c4003cb0 2021-03-09T09:52:54.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056630 con 0x7f21cc042f20 2021-03-09T09:52:54.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:54.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:54.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:54.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:54.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:54.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:54.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:54.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:54.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:54.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056770 con 0x7f21cc042f20 2021-03-09T09:52:54.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:52:54.842 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:54.797+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25112 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:54.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.797+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023fa0 secure :-1 s=READY pgs=7438 cs=0 l=1 rev1=1 rx=0x7f21cc0506f0 tx=0x7f21dc0835d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:54.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:52:54.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.801+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc042f20 2021-03-09T09:52:54.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:52:54.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.805+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:54.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.805+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:54.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:54.805+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:52:59.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:59.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:59.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023fa0 secure :-1 s=READY pgs=7438 cs=0 l=1 rev1=1 rx=0x7f21cc0506f0 tx=0x7f21dc0835d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:59.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023fa0 secure :-1 s=READY pgs=7438 cs=0 l=1 rev1=1 rx=0x7f21cc0506f0 tx=0x7f21dc0835d0).stop 2021-03-09T09:52:59.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:59.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023fa0 unknown :-1 s=CLOSED pgs=7438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:59.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:59.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:59.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:59.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d500 con 0x7f21cc005e30 2021-03-09T09:52:59.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f640 con 0x7f21c4003cb0 2021-03-09T09:52:59.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044810 con 0x7f21cc02f1a0 2021-03-09T09:52:59.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:52:59.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:59.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:52:59.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:59.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.797+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:59.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:59.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:59.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:59.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:59.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:52:59.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc02f1a0 2021-03-09T09:52:59.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7450 cs=0 l=1 rev1=1 rx=0x7f21cc0919c0 tx=0x7f21e4086070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:59.809 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:52:59.801+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25150 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:52:59.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21cc02f1a0 2021-03-09T09:52:59.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc02f1a0 2021-03-09T09:52:59.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.809+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc02f1a0 2021-03-09T09:52:59.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.809+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:52:59.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.813+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:52:59.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:52:59.813+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:04.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:04.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:04.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7450 cs=0 l=1 rev1=1 rx=0x7f21cc0919c0 tx=0x7f21e4086070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:04.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7450 cs=0 l=1 rev1=1 rx=0x7f21cc0919c0 tx=0x7f21e4086070).stop 2021-03-09T09:53:04.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:04.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:04.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:04.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:04.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:04.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050680 con 0x7f21cc005e30 2021-03-09T09:53:04.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044810 con 0x7f21c4003cb0 2021-03-09T09:53:04.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094530 con 0x7f21cc042f20 2021-03-09T09:53:04.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:04.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:04.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:04.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:04.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:04.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:04.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:04.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:04.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:04.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:53:04.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056770 con 0x7f21cc042f20 2021-03-09T09:53:04.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023c60 secure :-1 s=READY pgs=7460 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc085ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:04.815 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25185 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:04.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc042f20 2021-03-09T09:53:04.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.809+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc072ad0 con 0x7f21cc042f20 2021-03-09T09:53:04.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:53:04.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.813+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:04.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.813+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:04.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:04.813+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:09.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:53:09.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:09.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023c60 secure :-1 s=READY pgs=7460 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc085ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:09.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023c60 secure :-1 s=READY pgs=7460 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc085ee0).stop 2021-03-09T09:53:09.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:09.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023c60 unknown :-1 s=CLOSED pgs=7460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:09.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:09.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:09.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:09.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d500 con 0x7f21cc005e30 2021-03-09T09:53:09.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21c4003cb0 2021-03-09T09:53:09.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6a0 con 0x7f21cc02f1a0 2021-03-09T09:53:09.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:09.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:09.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:09.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:09.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:09.816 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25223 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:09.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:09.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:09.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:09.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.808+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:09.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089500 con 0x7f21cc02f1a0 2021-03-09T09:53:09.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc02f1a0 2021-03-09T09:53:09.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7475 cs=0 l=1 rev1=1 rx=0x7f21dc084010 tx=0x7f21dc055650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:09.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc09b220 con 0x7f21cc02f1a0 2021-03-09T09:53:09.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09b220 con 0x7f21cc02f1a0 2021-03-09T09:53:09.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc02f1a0 2021-03-09T09:53:09.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.820+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:09.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.820+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:09.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:09.820+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:14.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:14.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:14.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7475 cs=0 l=1 rev1=1 rx=0x7f21dc084010 tx=0x7f21dc055650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:14.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7475 cs=0 l=1 rev1=1 rx=0x7f21dc084010 tx=0x7f21dc055650).stop 2021-03-09T09:53:14.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:14.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:14.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:14.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:14.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:14.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d500 con 0x7f21cc005e30 2021-03-09T09:53:14.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf70 con 0x7f21c4003cb0 2021-03-09T09:53:14.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21cc042f20 2021-03-09T09:53:14.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:14.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:14.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:14.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:14.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:14.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:14.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:14.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:14.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:14.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f640 con 0x7f21cc042f20 2021-03-09T09:53:14.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089500 con 0x7f21cc042f20 2021-03-09T09:53:14.823 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:14.816+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25260 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:14.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7480 cs=0 l=1 rev1=1 rx=0x7f21cc00f800 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:14.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02a710 con 0x7f21cc042f20 2021-03-09T09:53:14.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc042f20 2021-03-09T09:53:14.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc042f20 2021-03-09T09:53:14.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.824+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:14.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:14.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:14.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:19.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:19.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:19.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7480 cs=0 l=1 rev1=1 rx=0x7f21cc00f800 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:19.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7480 cs=0 l=1 rev1=1 rx=0x7f21cc00f800 tx=0x7f21dc077e50).stop 2021-03-09T09:53:19.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:19.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:19.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:19.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:19.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:19.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d500 con 0x7f21cc005e30 2021-03-09T09:53:19.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f6c0 con 0x7f21c4003cb0 2021-03-09T09:53:19.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc024330 con 0x7f21cc02f1a0 2021-03-09T09:53:19.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:19.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:19.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:19.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:19.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:19.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:19.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:19.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:19.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:19.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094530 con 0x7f21cc02f1a0 2021-03-09T09:53:19.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.816+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f640 con 0x7f21cc02f1a0 2021-03-09T09:53:19.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7491 cs=0 l=1 rev1=1 rx=0x7f21dc055810 tx=0x7f21dc02a5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:19.827 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:19.820+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25297 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:19.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02a710 con 0x7f21cc02f1a0 2021-03-09T09:53:19.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02a710 con 0x7f21cc02f1a0 2021-03-09T09:53:19.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:53:19.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.828+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:19.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.828+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:19.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:19.828+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:22.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:22.778 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:24.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:24.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:24.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7491 cs=0 l=1 rev1=1 rx=0x7f21dc055810 tx=0x7f21dc02a5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:24.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7491 cs=0 l=1 rev1=1 rx=0x7f21dc055810 tx=0x7f21dc02a5b0).stop 2021-03-09T09:53:24.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:24.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:24.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:24.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:24.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:24.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ee40 con 0x7f21cc005e30 2021-03-09T09:53:24.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d500 con 0x7f21c4003cb0 2021-03-09T09:53:24.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae90 con 0x7f21cc042f20 2021-03-09T09:53:24.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:24.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:24.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:24.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:24.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:24.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:24.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:24.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:24.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:24.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089500 con 0x7f21cc042f20 2021-03-09T09:53:24.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094530 con 0x7f21cc042f20 2021-03-09T09:53:24.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7502 cs=0 l=1 rev1=1 rx=0x7f21dc062ce0 tx=0x7f21dc065430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:24.833 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:24.824+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25333 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:24.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:53:24.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02a710 con 0x7f21cc042f20 2021-03-09T09:53:24.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:53:24.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.832+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:24.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.832+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:24.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:24.832+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:29.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:29.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:29.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7502 cs=0 l=1 rev1=1 rx=0x7f21dc062ce0 tx=0x7f21dc065430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:29.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7502 cs=0 l=1 rev1=1 rx=0x7f21dc062ce0 tx=0x7f21dc065430).stop 2021-03-09T09:53:29.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:29.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:29.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:29.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:29.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:29.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21cc005e30 2021-03-09T09:53:29.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094530 con 0x7f21c4003cb0 2021-03-09T09:53:29.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0529a0 con 0x7f21cc02f1a0 2021-03-09T09:53:29.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:29.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:29.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:29.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:29.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:29.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:29.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:29.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:29.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:29.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc02f1a0 2021-03-09T09:53:29.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089500 con 0x7f21cc02f1a0 2021-03-09T09:53:29.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc096fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:29.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc02f1a0 2021-03-09T09:53:29.838 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:29.828+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25371 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:29.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005950 con 0x7f21cc02f1a0 2021-03-09T09:53:29.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:53:29.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.836+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:29.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.836+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:29.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:29.836+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:34.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:34.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:34.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc096fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:34.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc096fc0).stop 2021-03-09T09:53:34.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:34.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:34.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:34.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:34.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:34.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089520 con 0x7f21cc005e30 2021-03-09T09:53:34.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21c4003cb0 2021-03-09T09:53:34.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae90 con 0x7f21cc042f20 2021-03-09T09:53:34.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:34.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:34.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:34.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:34.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:34.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:34.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:34.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:34.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:34.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094530 con 0x7f21cc042f20 2021-03-09T09:53:34.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:53:34.842 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:34.832+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25407 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:34.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7526 cs=0 l=1 rev1=1 rx=0x7f21dc0334b0 tx=0x7f21dc0655b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:34.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc042f20 2021-03-09T09:53:34.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc042f20 2021-03-09T09:53:34.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc042f20 2021-03-09T09:53:34.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.844+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:34.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:34.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:34.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:39.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:39.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:39.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7526 cs=0 l=1 rev1=1 rx=0x7f21dc0334b0 tx=0x7f21dc0655b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:39.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 secure :-1 s=READY pgs=7526 cs=0 l=1 rev1=1 rx=0x7f21dc0334b0 tx=0x7f21dc0655b0).stop 2021-03-09T09:53:39.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:39.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:39.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:39.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:39.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:39.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21cc005e30 2021-03-09T09:53:39.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01da80 con 0x7f21c4003cb0 2021-03-09T09:53:39.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ed0 con 0x7f21cc02f1a0 2021-03-09T09:53:39.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:39.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:39.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:39.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:39.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:39.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:39.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:39.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:39.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:39.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ae90 con 0x7f21cc02f1a0 2021-03-09T09:53:39.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094530 con 0x7f21cc02f1a0 2021-03-09T09:53:39.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7534 cs=0 l=1 rev1=1 rx=0x7f21dc095c30 tx=0x7f21dc062f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:39.847 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:39.840+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25445 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:39.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099080 con 0x7f21cc02f1a0 2021-03-09T09:53:39.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc019af0 con 0x7f21cc02f1a0 2021-03-09T09:53:39.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:53:39.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.844+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:39.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:39.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:39.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:44.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:44.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:44.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7534 cs=0 l=1 rev1=1 rx=0x7f21dc095c30 tx=0x7f21dc062f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:44.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7534 cs=0 l=1 rev1=1 rx=0x7f21dc095c30 tx=0x7f21dc062f90).stop 2021-03-09T09:53:44.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:44.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:44.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:44.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:44.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:44.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc005e30 2021-03-09T09:53:44.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21c4003cb0 2021-03-09T09:53:44.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01da60 con 0x7f21cc042f20 2021-03-09T09:53:44.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:44.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:44.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:44.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:44.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:44.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:44.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:44.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:44.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:44.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:53:44.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:53:44.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f320 secure :-1 s=READY pgs=7544 cs=0 l=1 rev1=1 rx=0x7f21cc058f90 tx=0x7f21e4016c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:44.859 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:44.844+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25481 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:44.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404a070 con 0x7f21cc042f20 2021-03-09T09:53:44.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4015d80 con 0x7f21cc042f20 2021-03-09T09:53:44.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404a070 con 0x7f21cc042f20 2021-03-09T09:53:44.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.856+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:44.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.856+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:44.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:44.856+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:49.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:53:49.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:49.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f320 secure :-1 s=READY pgs=7544 cs=0 l=1 rev1=1 rx=0x7f21cc058f90 tx=0x7f21e4016c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:49.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f320 secure :-1 s=READY pgs=7544 cs=0 l=1 rev1=1 rx=0x7f21cc058f90 tx=0x7f21e4016c00).stop 2021-03-09T09:53:49.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:49.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f320 unknown :-1 s=CLOSED pgs=7544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:49.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:49.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc005e30 2021-03-09T09:53:49.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21c4003cb0 2021-03-09T09:53:49.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01da60 con 0x7f21cc02f1a0 2021-03-09T09:53:49.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:49.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:49.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:49.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:49.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:49.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:49.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:49.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:49.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:49.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:53:49.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ae90 con 0x7f21cc02f1a0 2021-03-09T09:53:49.858 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25519 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:49.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.848+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7556 cs=0 l=1 rev1=1 rx=0x7f21dc00f040 tx=0x7f21dc0b6400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:49.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004370 con 0x7f21cc02f1a0 2021-03-09T09:53:49.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T09:53:49.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:53:49.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.856+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:49.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.856+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:49.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:49.856+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:52.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:52.851 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:53:54.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:54.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:54.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7556 cs=0 l=1 rev1=1 rx=0x7f21dc00f040 tx=0x7f21dc0b6400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:54.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7556 cs=0 l=1 rev1=1 rx=0x7f21dc00f040 tx=0x7f21dc0b6400).stop 2021-03-09T09:53:54.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:54.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:54.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:54.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:54.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:54.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc005e30 2021-03-09T09:53:54.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035cd0 con 0x7f21c4003cb0 2021-03-09T09:53:54.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21cc042f20 2021-03-09T09:53:54.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:54.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:54.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:54.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:54.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:54.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:54.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:54.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:54.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:54.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ae90 con 0x7f21cc042f20 2021-03-09T09:53:54.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:53:54.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7567 cs=0 l=1 rev1=1 rx=0x7f21e4053f80 tx=0x7f21e40a5350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:54.861 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:54.852+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25555 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:54.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400dc70 con 0x7f21cc042f20 2021-03-09T09:53:54.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc042f20 2021-03-09T09:53:54.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.864+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400dc70 con 0x7f21cc042f20 2021-03-09T09:53:54.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.864+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:54.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.864+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:54.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:54.864+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:53:59.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:53:59.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:59.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7567 cs=0 l=1 rev1=1 rx=0x7f21e4053f80 tx=0x7f21e40a5350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:59.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7567 cs=0 l=1 rev1=1 rx=0x7f21e4053f80 tx=0x7f21e40a5350).stop 2021-03-09T09:53:59.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:59.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=CLOSED pgs=7567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:59.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:59.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:59.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:59.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc005e30 2021-03-09T09:53:59.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21c4003cb0 2021-03-09T09:53:59.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01da60 con 0x7f21cc02f1a0 2021-03-09T09:53:59.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:53:59.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:59.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:53:59.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:59.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:59.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:59.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:59.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:59.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:59.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc009b40 con 0x7f21cc02f1a0 2021-03-09T09:53:59.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:53:59.869 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:53:59.855+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25593 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:53:59.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7579 cs=0 l=1 rev1=1 rx=0x7f21dc07cca0 tx=0x7f21dc080080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:59.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0568d0 con 0x7f21cc02f1a0 2021-03-09T09:53:59.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T09:53:59.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d30 con 0x7f21cc02f1a0 2021-03-09T09:53:59.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.863+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:53:59.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.867+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:53:59.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:53:59.867+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:04.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:04.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:04.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7579 cs=0 l=1 rev1=1 rx=0x7f21dc07cca0 tx=0x7f21dc080080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:04.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7579 cs=0 l=1 rev1=1 rx=0x7f21dc07cca0 tx=0x7f21dc080080).stop 2021-03-09T09:54:04.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:04.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:04.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:04.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:04.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:04.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc005e30 2021-03-09T09:54:04.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21c4003cb0 2021-03-09T09:54:04.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:54:04.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:04.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:04.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:04.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:04.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:04.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:04.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:04.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:04.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:04.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc035c80 con 0x7f21cc042f20 2021-03-09T09:54:04.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.859+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc009b40 con 0x7f21cc042f20 2021-03-09T09:54:04.871 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:04.863+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25630 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:04.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.863+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7587 cs=0 l=1 rev1=1 rx=0x7f21cc089550 tx=0x7f21dc056d70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:04.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc029ce0 con 0x7f21cc042f20 2021-03-09T09:54:04.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc042f20 2021-03-09T09:54:04.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc077fc0 con 0x7f21cc042f20 2021-03-09T09:54:04.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.867+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:04.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.867+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:04.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:04.867+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:09.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:09.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:09.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7587 cs=0 l=1 rev1=1 rx=0x7f21cc089550 tx=0x7f21dc056d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:09.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7587 cs=0 l=1 rev1=1 rx=0x7f21cc089550 tx=0x7f21dc056d70).stop 2021-03-09T09:54:09.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:09.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=CLOSED pgs=7587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:09.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:09.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:09.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:09.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21cc005e30 2021-03-09T09:54:09.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4003cb0 2021-03-09T09:54:09.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0319c0 con 0x7f21cc02f1a0 2021-03-09T09:54:09.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:09.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:09.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:09.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:09.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:09.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:09.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:09.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.867+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:09.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:09.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023ac0 con 0x7f21cc02f1a0 2021-03-09T09:54:09.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ae90 con 0x7f21cc02f1a0 2021-03-09T09:54:09.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7598 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4009a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:09.876 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25667 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:09.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T09:54:09.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21cc02f1a0 2021-03-09T09:54:09.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T09:54:09.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.875+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:09.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.875+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:09.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:09.875+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:14.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:14.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:14.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7598 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4009a80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:14.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7598 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4009a80).stop 2021-03-09T09:54:14.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:14.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:14.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:14.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:14.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:14.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e210 con 0x7f21cc005e30 2021-03-09T09:54:14.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b090 con 0x7f21c4003cb0 2021-03-09T09:54:14.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0319c0 con 0x7f21cc042f20 2021-03-09T09:54:14.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:14.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:14.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:14.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:14.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:14.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:14.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:14.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:14.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:14.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:54:14.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.871+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc035c80 con 0x7f21cc042f20 2021-03-09T09:54:14.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7608 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0b6120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:14.880 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:14.875+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25703 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:14.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc042f20 2021-03-09T09:54:14.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02aea0 con 0x7f21cc042f20 2021-03-09T09:54:14.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:54:14.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.879+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:14.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.879+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:14.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:14.879+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:19.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:19.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:19.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7608 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0b6120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:19.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7608 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0b6120).stop 2021-03-09T09:54:19.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:19.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=CLOSED pgs=7608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:19.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:19.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:19.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:19.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031680 con 0x7f21cc005e30 2021-03-09T09:54:19.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0317c0 con 0x7f21c4003cb0 2021-03-09T09:54:19.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031900 con 0x7f21cc02f1a0 2021-03-09T09:54:19.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:19.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:19.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:19.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:19.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:19.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:19.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:19.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:19.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:19.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031a70 con 0x7f21cc02f1a0 2021-03-09T09:54:19.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023ac0 con 0x7f21cc02f1a0 2021-03-09T09:54:19.885 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:19.879+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25741 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:19.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.883+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7620 cs=0 l=1 rev1=1 rx=0x7f21cc089550 tx=0x7f21e40557f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:19.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21cc02f1a0 2021-03-09T09:54:19.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T09:54:19.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T09:54:19.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.887+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:19.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.891+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:19.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:19.891+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:22.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:22.919 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:24.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:54:24.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:24.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7620 cs=0 l=1 rev1=1 rx=0x7f21cc089550 tx=0x7f21e40557f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:24.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7620 cs=0 l=1 rev1=1 rx=0x7f21cc089550 tx=0x7f21e40557f0).stop 2021-03-09T09:54:24.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:24.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:24.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:24.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:24.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:24.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009950 con 0x7f21cc005e30 2021-03-09T09:54:24.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031630 con 0x7f21c4003cb0 2021-03-09T09:54:24.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031770 con 0x7f21cc042f20 2021-03-09T09:54:24.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.879+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:24.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:24.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:24.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:24.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:24.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:24.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:24.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:24.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:24.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0318e0 con 0x7f21cc042f20 2021-03-09T09:54:24.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:54:24.890 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:24.883+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25777 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:24.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7633 cs=0 l=1 rev1=1 rx=0x7f21cc094e50 tx=0x7f21dc050ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:24.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06a5b0 con 0x7f21cc042f20 2021-03-09T09:54:24.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06a5b0 con 0x7f21cc042f20 2021-03-09T09:54:24.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:54:24.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.891+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:24.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.895+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:24.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:24.895+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:29.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:29.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:29.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7633 cs=0 l=1 rev1=1 rx=0x7f21cc094e50 tx=0x7f21dc050ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:29.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7633 cs=0 l=1 rev1=1 rx=0x7f21cc094e50 tx=0x7f21dc050ea0).stop 2021-03-09T09:54:29.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:29.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=CLOSED pgs=7633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:29.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:29.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:29.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:29.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009950 con 0x7f21cc005e30 2021-03-09T09:54:29.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d170 con 0x7f21c4003cb0 2021-03-09T09:54:29.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:54:29.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:29.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:29.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:29.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:29.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.887+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:29.893 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25815 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:29.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:29.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:29.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:29.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:29.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023ac0 con 0x7f21cc02f1a0 2021-03-09T09:54:29.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031a70 con 0x7f21cc02f1a0 2021-03-09T09:54:29.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7646 cs=0 l=1 rev1=1 rx=0x7f21e407d250 tx=0x7f21e40860e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:29.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5160 con 0x7f21cc02f1a0 2021-03-09T09:54:29.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007390 con 0x7f21cc02f1a0 2021-03-09T09:54:29.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc02f1a0 2021-03-09T09:54:29.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.895+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:29.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.899+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:29.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:29.899+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:34.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:34.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:34.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7646 cs=0 l=1 rev1=1 rx=0x7f21e407d250 tx=0x7f21e40860e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:34.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7646 cs=0 l=1 rev1=1 rx=0x7f21e407d250 tx=0x7f21e40860e0).stop 2021-03-09T09:54:34.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:34.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:34.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:34.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:34.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:34.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009950 con 0x7f21cc005e30 2021-03-09T09:54:34.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015600 con 0x7f21c4003cb0 2021-03-09T09:54:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc042f20 2021-03-09T09:54:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:34.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:34.898 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25852 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:34.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:34.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:34.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:34.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:34.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031a70 con 0x7f21cc042f20 2021-03-09T09:54:34.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0318e0 con 0x7f21cc042f20 2021-03-09T09:54:34.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7656 cs=0 l=1 rev1=1 rx=0x7f21dc051c10 tx=0x7f21dc050e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:34.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21cc042f20 2021-03-09T09:54:34.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc042f20 2021-03-09T09:54:34.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:54:34.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.903+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:34.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:34.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:34.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:39.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:39.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7656 cs=0 l=1 rev1=1 rx=0x7f21dc051c10 tx=0x7f21dc050e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7656 cs=0 l=1 rev1=1 rx=0x7f21dc051c10 tx=0x7f21dc050e30).stop 2021-03-09T09:54:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=CLOSED pgs=7656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009950 con 0x7f21cc005e30 2021-03-09T09:54:39.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0155d0 con 0x7f21c4003cb0 2021-03-09T09:54:39.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:54:39.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:39.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:39.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:39.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:39.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:39.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:39.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:39.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:39.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.895+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:39.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.899+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0318e0 con 0x7f21cc02f1a0 2021-03-09T09:54:39.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.899+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023ac0 con 0x7f21cc02f1a0 2021-03-09T09:54:39.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.899+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7662 cs=0 l=1 rev1=1 rx=0x7f21cc094e50 tx=0x7f21e407d290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:39.905 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:39.899+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25889 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:39.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4086580 con 0x7f21cc02f1a0 2021-03-09T09:54:39.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402bea0 con 0x7f21cc02f1a0 2021-03-09T09:54:39.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028d40 con 0x7f21cc02f1a0 2021-03-09T09:54:39.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.903+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:39.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:39.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:39.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:44.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:44.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:44.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7662 cs=0 l=1 rev1=1 rx=0x7f21cc094e50 tx=0x7f21e407d290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:44.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7662 cs=0 l=1 rev1=1 rx=0x7f21cc094e50 tx=0x7f21e407d290).stop 2021-03-09T09:54:44.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:44.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:44.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054290 con 0x7f21cc005e30 2021-03-09T09:54:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4003cb0 2021-03-09T09:54:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039d30 con 0x7f21cc042f20 2021-03-09T09:54:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.899+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:44.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:44.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039e70 con 0x7f21cc042f20 2021-03-09T09:54:44.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031a70 con 0x7f21cc042f20 2021-03-09T09:54:44.910 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:44.903+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25926 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:44.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.907+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7676 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0ba000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:44.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.907+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc042f20 2021-03-09T09:54:44.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050ea0 con 0x7f21cc042f20 2021-03-09T09:54:44.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T09:54:44.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.915+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:44.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.915+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:44.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:44.915+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:49.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:49.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:49.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7676 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0ba000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:49.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 secure :-1 s=READY pgs=7676 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc0ba000).stop 2021-03-09T09:54:49.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc03f210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:49.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc03f210 unknown :-1 s=CLOSED pgs=7676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:49.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:49.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:49.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:49.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a000 con 0x7f21cc005e30 2021-03-09T09:54:49.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039d10 con 0x7f21c4003cb0 2021-03-09T09:54:49.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:54:49.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:49.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:49.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:49.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:49.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:49.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:49.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:49.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:49.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:49.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031a70 con 0x7f21cc02f1a0 2021-03-09T09:54:49.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039e70 con 0x7f21cc02f1a0 2021-03-09T09:54:49.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7683 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc018330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:49.914 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:49.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25963 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:49.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc02f1a0 2021-03-09T09:54:49.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050ea0 con 0x7f21cc02f1a0 2021-03-09T09:54:49.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc02f1a0 2021-03-09T09:54:49.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.914+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:49.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:49.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:49.914+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:51.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:54:51.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:51.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 0x7f21cc0048d0 secure :-1 s=READY pgs=5498 cs=0 l=1 rev1=1 rx=0x7f21e0005450 tx=0x7f21e001f730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:51.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 0x7f21cc0048d0 secure :-1 s=READY pgs=5498 cs=0 l=1 rev1=1 rx=0x7f21e0005450 tx=0x7f21e001f730).stop 2021-03-09T09:54:51.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:54:51.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21c4008860 2021-03-09T09:54:51.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T09:54:51.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:51.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4008860 0x7f21cc0048d0 unknown :-1 s=CLOSED pgs=5498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:51.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:51.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:51.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 0x7f21cc039b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:51.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:51.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 0x7f21cc039b70 secure :-1 s=READY pgs=7307 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00b0000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:52.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:52.983 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:54:54.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:54.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:54.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7683 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc018330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:54.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7683 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc018330).stop 2021-03-09T09:54:54.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:54.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:54.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:54.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:54.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:54.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:54:54.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023ac0 con 0x7f21c4003cb0 2021-03-09T09:54:54.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023c00 con 0x7f21c4008860 2021-03-09T09:54:54.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:54.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:54.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:54.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:54.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:54.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:54.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:54.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:54.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:54.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023d40 con 0x7f21c4008860 2021-03-09T09:54:54.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031a70 con 0x7f21c4008860 2021-03-09T09:54:54.926 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25999 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:54.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048d0 secure :-1 s=READY pgs=7695 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc081210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21c4008860 2021-03-09T09:54:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21c4008860 2021-03-09T09:54:54.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21c4008860 2021-03-09T09:54:54.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.922+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:54.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:54.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:54.922+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:54:59.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:59.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:59.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048d0 secure :-1 s=READY pgs=7695 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc081210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:59.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048d0 secure :-1 s=READY pgs=7695 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc081210).stop 2021-03-09T09:54:59.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0048d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:59.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048d0 unknown :-1 s=CLOSED pgs=7695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:59.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:59.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:59.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:59.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032050 con 0x7f21cc005e30 2021-03-09T09:54:59.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057560 con 0x7f21c4003cb0 2021-03-09T09:54:59.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0527e0 con 0x7f21cc02f1a0 2021-03-09T09:54:59.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:54:59.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:59.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:54:59.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:59.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.914+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:59.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:59.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:59.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:59.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:59.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:54:59.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023d40 con 0x7f21cc02f1a0 2021-03-09T09:54:59.924 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26037 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:54:59.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7707 cs=0 l=1 rev1=1 rx=0x7f21dc0b6210 tx=0x7f21dc018610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:59.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc02f1a0 2021-03-09T09:54:59.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T09:54:59.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc02f1a0 2021-03-09T09:54:59.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:54:59.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:54:59.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:54:59.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:04.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:04.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:04.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7707 cs=0 l=1 rev1=1 rx=0x7f21dc0b6210 tx=0x7f21dc018610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:04.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 secure :-1 s=READY pgs=7707 cs=0 l=1 rev1=1 rx=0x7f21dc0b6210 tx=0x7f21dc018610).stop 2021-03-09T09:55:04.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0860c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:04.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0860c0 unknown :-1 s=CLOSED pgs=7707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:04.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:04.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:04.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:04.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057560 con 0x7f21cc005e30 2021-03-09T09:55:04.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013210 con 0x7f21c4003cb0 2021-03-09T09:55:04.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004da0 con 0x7f21c4008860 2021-03-09T09:55:04.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:04.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:04.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:04.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.918+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:04.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:04.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:04.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:04.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:04.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:04.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031a70 con 0x7f21c4008860 2021-03-09T09:55:04.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:55:04.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048b0 secure :-1 s=READY pgs=7715 cs=0 l=1 rev1=1 rx=0x7f21dc0813d0 tx=0x7f21dc080040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:04.930 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:04.922+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26072 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:04.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T09:55:04.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T09:55:04.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21c4008860 2021-03-09T09:55:04.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.930+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:04.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:04.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:04.930+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:09.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:09.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:09.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048b0 secure :-1 s=READY pgs=7715 cs=0 l=1 rev1=1 rx=0x7f21dc0813d0 tx=0x7f21dc080040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:09.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048b0 secure :-1 s=READY pgs=7715 cs=0 l=1 rev1=1 rx=0x7f21dc0813d0 tx=0x7f21dc080040).stop 2021-03-09T09:55:09.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0048b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:09.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0048b0 unknown :-1 s=CLOSED pgs=7715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:09.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:09.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:09.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:09.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc005e30 2021-03-09T09:55:09.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0860c0 con 0x7f21c4003cb0 2021-03-09T09:55:09.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086200 con 0x7f21cc02f1a0 2021-03-09T09:55:09.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:09.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:09.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:09.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:09.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:09.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:09.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:09.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:09.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:09.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086340 con 0x7f21cc02f1a0 2021-03-09T09:55:09.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031a70 con 0x7f21cc02f1a0 2021-03-09T09:55:09.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7726 cs=0 l=1 rev1=1 rx=0x7f21dc005410 tx=0x7f21dc08c7d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:09.935 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:09.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26111 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:09.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21cc02f1a0 2021-03-09T09:55:09.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085e70 con 0x7f21cc02f1a0 2021-03-09T09:55:09.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc02f1a0 2021-03-09T09:55:09.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:09.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:09.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:09.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:14.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:14.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:14.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7726 cs=0 l=1 rev1=1 rx=0x7f21dc005410 tx=0x7f21dc08c7d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:14.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7726 cs=0 l=1 rev1=1 rx=0x7f21dc005410 tx=0x7f21dc08c7d0).stop 2021-03-09T09:55:14.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:14.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=CLOSED pgs=7726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:14.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:14.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:14.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:14.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04f840 con 0x7f21cc005e30 2021-03-09T09:55:14.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0864d0 con 0x7f21c4003cb0 2021-03-09T09:55:14.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:55:14.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:14.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:14.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:14.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:14.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:14.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:14.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:14.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:14.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:14.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031a70 con 0x7f21c4008860 2021-03-09T09:55:14.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0318e0 con 0x7f21c4008860 2021-03-09T09:55:14.940 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26147 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:14.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7739 cs=0 l=1 rev1=1 rx=0x7f21cc056440 tx=0x7f21e4016f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:14.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21c4008860 2021-03-09T09:55:14.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21c4008860 2021-03-09T09:55:14.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028c60 con 0x7f21c4008860 2021-03-09T09:55:14.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.942+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:14.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.942+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:14.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:14.942+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:19.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:19.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:19.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7739 cs=0 l=1 rev1=1 rx=0x7f21cc056440 tx=0x7f21e4016f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:19.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7739 cs=0 l=1 rev1=1 rx=0x7f21cc056440 tx=0x7f21e4016f10).stop 2021-03-09T09:55:19.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:19.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:19.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:19.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:19.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:19.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0864d0 con 0x7f21cc005e30 2021-03-09T09:55:19.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275e0 con 0x7f21c4003cb0 2021-03-09T09:55:19.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:55:19.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:19.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:19.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:19.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:19.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.934+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:19.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:19.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:19.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:19.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:19.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04f840 con 0x7f21cc02f1a0 2021-03-09T09:55:19.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086340 con 0x7f21cc02f1a0 2021-03-09T09:55:19.946 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26185 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:19.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7750 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc09bfa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:19.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05b000 con 0x7f21cc02f1a0 2021-03-09T09:55:19.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21cc02f1a0 2021-03-09T09:55:19.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21cc02f1a0 2021-03-09T09:55:19.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.946+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:19.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:19.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:19.946+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:23.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:23.026 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:24.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:24.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:24.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7750 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc09bfa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:24.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7750 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc09bfa0).stop 2021-03-09T09:55:24.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:24.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=CLOSED pgs=7750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:24.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:24.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:24.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:24.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275e0 con 0x7f21cc005e30 2021-03-09T09:55:24.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0918c0 con 0x7f21c4003cb0 2021-03-09T09:55:24.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094200 con 0x7f21c4008860 2021-03-09T09:55:24.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:24.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:24.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:24.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.938+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:24.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:24.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:24.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:24.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:24.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:24.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:55:24.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031a70 con 0x7f21c4008860 2021-03-09T09:55:24.949 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:24.942+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26221 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:24.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012e50 secure :-1 s=READY pgs=7761 cs=0 l=1 rev1=1 rx=0x7f21cc03fc20 tx=0x7f21e4009b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:24.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407fdd0 con 0x7f21c4008860 2021-03-09T09:55:24.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21c4008860 2021-03-09T09:55:24.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T09:55:24.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.950+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:24.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:24.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:24.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:29.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:29.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:29.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012e50 secure :-1 s=READY pgs=7761 cs=0 l=1 rev1=1 rx=0x7f21cc03fc20 tx=0x7f21e4009b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:29.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012e50 secure :-1 s=READY pgs=7761 cs=0 l=1 rev1=1 rx=0x7f21cc03fc20 tx=0x7f21e4009b00).stop 2021-03-09T09:55:29.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:29.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012e50 unknown :-1 s=CLOSED pgs=7761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:29.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:29.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:29.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:29.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094200 con 0x7f21cc005e30 2021-03-09T09:55:29.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027590 con 0x7f21c4003cb0 2021-03-09T09:55:29.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0860c0 con 0x7f21cc02f1a0 2021-03-09T09:55:29.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:29.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:29.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:29.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:29.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:29.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:29.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:29.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:29.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:29.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091720 con 0x7f21cc02f1a0 2021-03-09T09:55:29.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:55:29.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7770 cs=0 l=1 rev1=1 rx=0x7f21cc04ded0 tx=0x7f21e404fd50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:29.956 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:29.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26259 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:29.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003af0 con 0x7f21cc02f1a0 2021-03-09T09:55:29.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T09:55:29.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4048180 con 0x7f21cc02f1a0 2021-03-09T09:55:29.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.954+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:29.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:29.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:29.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:34.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:34.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:34.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7770 cs=0 l=1 rev1=1 rx=0x7f21cc04ded0 tx=0x7f21e404fd50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:34.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 secure :-1 s=READY pgs=7770 cs=0 l=1 rev1=1 rx=0x7f21cc04ded0 tx=0x7f21e404fd50).stop 2021-03-09T09:55:34.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:34.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095190 unknown :-1 s=CLOSED pgs=7770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:34.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:34.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:34.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:34.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275d0 con 0x7f21cc005e30 2021-03-09T09:55:34.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4003cb0 2021-03-09T09:55:34.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023fb0 con 0x7f21c4008860 2021-03-09T09:55:34.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:34.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:34.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:34.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:34.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:34.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:34.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:34.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:34.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:34.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093fc0 con 0x7f21c4008860 2021-03-09T09:55:34.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091720 con 0x7f21c4008860 2021-03-09T09:55:34.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.950+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7779 cs=0 l=1 rev1=1 rx=0x7f21cc093e30 tx=0x7f21e4020fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:34.960 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:34.954+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26295 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:34.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a4aa0 con 0x7f21c4008860 2021-03-09T09:55:34.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a4cf0 con 0x7f21c4008860 2021-03-09T09:55:34.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4008860 2021-03-09T09:55:34.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:34.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:34.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:34.958+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:39.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:39.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:39.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7779 cs=0 l=1 rev1=1 rx=0x7f21cc093e30 tx=0x7f21e4020fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:39.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7779 cs=0 l=1 rev1=1 rx=0x7f21cc093e30 tx=0x7f21e4020fd0).stop 2021-03-09T09:55:39.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:39.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:39.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:39.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:39.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:39.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0275d0 con 0x7f21cc005e30 2021-03-09T09:55:39.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023fb0 con 0x7f21c4003cb0 2021-03-09T09:55:39.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:55:39.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:39.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:39.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:39.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:39.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:39.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:39.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:39.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:39.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:39.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086070 con 0x7f21cc02f1a0 2021-03-09T09:55:39.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093fc0 con 0x7f21cc02f1a0 2021-03-09T09:55:39.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.957+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095140 secure :-1 s=READY pgs=7790 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4017ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:39.965 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:39.957+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26330 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:39.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc02f1a0 2021-03-09T09:55:39.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21cc02f1a0 2021-03-09T09:55:39.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc02f1a0 2021-03-09T09:55:39.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.961+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:39.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:39.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:39.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:44.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:44.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:44.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095140 secure :-1 s=READY pgs=7790 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4017ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:44.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095140 secure :-1 s=READY pgs=7790 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4017ec0).stop 2021-03-09T09:55:44.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc095140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:44.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc095140 unknown :-1 s=CLOSED pgs=7790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:44.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:44.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:44.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:44.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:44.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023fb0 con 0x7f21cc005e30 2021-03-09T09:55:44.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:44.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4003cb0 2021-03-09T09:55:44.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031890 con 0x7f21c4008860 2021-03-09T09:55:44.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:44.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:44.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.957+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:44.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:44.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:44.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:44.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:44.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0319d0 con 0x7f21c4008860 2021-03-09T09:55:44.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086070 con 0x7f21c4008860 2021-03-09T09:55:44.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7800 cs=0 l=1 rev1=1 rx=0x7f21e402b570 tx=0x7f21e409bc80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:44.976 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:44.961+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26369 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:44.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21c4008860 2021-03-09T09:55:44.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a4aa0 con 0x7f21c4008860 2021-03-09T09:55:44.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T09:55:44.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.969+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:44.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.969+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:44.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:44.969+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:49.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:49.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:49.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7800 cs=0 l=1 rev1=1 rx=0x7f21e402b570 tx=0x7f21e409bc80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:49.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7800 cs=0 l=1 rev1=1 rx=0x7f21e402b570 tx=0x7f21e409bc80).stop 2021-03-09T09:55:49.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:49.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:49.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:49.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:49.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:49.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023fb0 con 0x7f21cc005e30 2021-03-09T09:55:49.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026920 con 0x7f21c4003cb0 2021-03-09T09:55:49.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031890 con 0x7f21cc02f1a0 2021-03-09T09:55:49.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:49.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:49.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:49.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:49.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:49.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:49.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:49.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:49.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:49.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:55:49.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0319d0 con 0x7f21cc02f1a0 2021-03-09T09:55:49.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.965+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012be0 secure :-1 s=READY pgs=7811 cs=0 l=1 rev1=1 rx=0x7f21e401d060 tx=0x7f21e401ab50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:49.975 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:49.969+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26405 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:49.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc02f1a0 2021-03-09T09:55:49.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4092da0 con 0x7f21cc02f1a0 2021-03-09T09:55:49.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a4cf0 con 0x7f21cc02f1a0 2021-03-09T09:55:49.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.973+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:49.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.973+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:49.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:49.973+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:53.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:53.068 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:55:54.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:54.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:54.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012be0 secure :-1 s=READY pgs=7811 cs=0 l=1 rev1=1 rx=0x7f21e401d060 tx=0x7f21e401ab50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:54.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012be0 secure :-1 s=READY pgs=7811 cs=0 l=1 rev1=1 rx=0x7f21e401d060 tx=0x7f21e401ab50).stop 2021-03-09T09:55:54.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:54.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=7811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:54.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:54.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:54.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:54.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023710 con 0x7f21cc005e30 2021-03-09T09:55:54.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023fb0 con 0x7f21c4003cb0 2021-03-09T09:55:54.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013270 con 0x7f21c4008860 2021-03-09T09:55:54.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:54.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:54.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:54.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:54.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:54.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:54.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:54.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:54.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:54.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015760 con 0x7f21c4008860 2021-03-09T09:55:54.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04f840 con 0x7f21c4008860 2021-03-09T09:55:54.980 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26443 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:54.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7824 cs=0 l=1 rev1=1 rx=0x7f21cc044c40 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:54.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062ec0 con 0x7f21c4008860 2021-03-09T09:55:54.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21c4008860 2021-03-09T09:55:54.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T09:55:54.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.981+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:54.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:54.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:54.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:55:59.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:59.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:59.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7824 cs=0 l=1 rev1=1 rx=0x7f21cc044c40 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:59.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 secure :-1 s=READY pgs=7824 cs=0 l=1 rev1=1 rx=0x7f21cc044c40 tx=0x7f21dc077e50).stop 2021-03-09T09:55:59.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:59.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc023ac0 unknown :-1 s=CLOSED pgs=7824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:59.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:59.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:59.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:59.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dc00 con 0x7f21cc005e30 2021-03-09T09:55:59.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01da00 con 0x7f21c4003cb0 2021-03-09T09:55:59.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031a30 con 0x7f21cc02f1a0 2021-03-09T09:55:59.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:55:59.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:59.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:55:59.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:59.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.973+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:59.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:59.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:59.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:59.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:59.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086070 con 0x7f21cc02f1a0 2021-03-09T09:55:59.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:55:59.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0128a0 secure :-1 s=READY pgs=7832 cs=0 l=1 rev1=1 rx=0x7f21cc02d4b0 tx=0x7f21e40b0ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:59.986 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:55:59.977+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26479 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:55:59.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40059b0 con 0x7f21cc02f1a0 2021-03-09T09:55:59.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40059b0 con 0x7f21cc02f1a0 2021-03-09T09:55:59.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402d230 con 0x7f21cc02f1a0 2021-03-09T09:55:59.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.985+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:55:59.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.985+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:55:59.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:55:59.985+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:04.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:04.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:04.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0128a0 secure :-1 s=READY pgs=7832 cs=0 l=1 rev1=1 rx=0x7f21cc02d4b0 tx=0x7f21e40b0ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:04.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0128a0 secure :-1 s=READY pgs=7832 cs=0 l=1 rev1=1 rx=0x7f21cc02d4b0 tx=0x7f21e40b0ab0).stop 2021-03-09T09:56:04.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0128a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:04.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0128a0 unknown :-1 s=CLOSED pgs=7832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:04.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:04.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:04.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:04.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03dc00 con 0x7f21cc005e30 2021-03-09T09:56:04.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21c4003cb0 2021-03-09T09:56:04.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031a30 con 0x7f21c4008860 2021-03-09T09:56:04.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:04.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:04.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:04.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:04.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:04.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:04.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:04.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:04.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:04.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086070 con 0x7f21c4008860 2021-03-09T09:56:04.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015760 con 0x7f21c4008860 2021-03-09T09:56:04.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.981+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012d90 secure :-1 s=READY pgs=7843 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc056120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:04.992 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:04.985+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26517 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:04.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4008860 2021-03-09T09:56:04.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4008860 2021-03-09T09:56:04.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T09:56:04.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.989+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:04.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.989+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:04.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:04.989+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:09.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:09.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:09.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012d90 secure :-1 s=READY pgs=7843 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc056120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:09.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012d90 secure :-1 s=READY pgs=7843 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc056120).stop 2021-03-09T09:56:09.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:09.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012d90 unknown :-1 s=CLOSED pgs=7843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:09.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:09.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:09.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:09.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093d40 con 0x7f21cc005e30 2021-03-09T09:56:09.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ce20 con 0x7f21c4003cb0 2021-03-09T09:56:09.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1c0 con 0x7f21cc02f1a0 2021-03-09T09:56:09.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:09.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:09.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:09.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:09.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:09.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:09.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:09.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:09.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:09.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04da10 con 0x7f21cc02f1a0 2021-03-09T09:56:09.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:56:09.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.985+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012560 secure :-1 s=READY pgs=7854 cs=0 l=1 rev1=1 rx=0x7f21cc057b10 tx=0x7f21e4016b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:09.996 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:09.989+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:09.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc02f1a0 2021-03-09T09:56:09.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402db00 con 0x7f21cc02f1a0 2021-03-09T09:56:09.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.993+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402db00 con 0x7f21cc02f1a0 2021-03-09T09:56:09.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.993+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:09.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.997+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:09.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:09.997+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:14.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:14.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:14.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012560 secure :-1 s=READY pgs=7854 cs=0 l=1 rev1=1 rx=0x7f21cc057b10 tx=0x7f21e4016b60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:14.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012560 secure :-1 s=READY pgs=7854 cs=0 l=1 rev1=1 rx=0x7f21cc057b10 tx=0x7f21e4016b60).stop 2021-03-09T09:56:14.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:14.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012560 unknown :-1 s=CLOSED pgs=7854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:14.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:14.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:14.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:14.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ce20 con 0x7f21cc005e30 2021-03-09T09:56:14.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1c0 con 0x7f21c4003cb0 2021-03-09T09:56:14.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093d40 con 0x7f21c4008860 2021-03-09T09:56:14.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:14.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:14.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:14.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:14.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.989+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:15.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:15.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:15.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:15.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:15.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:56:15.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015760 con 0x7f21c4008860 2021-03-09T09:56:15.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012a50 secure :-1 s=READY pgs=7865 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0abf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:15.002 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:14.993+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26591 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:15.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0726b0 con 0x7f21c4008860 2021-03-09T09:56:15.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:14.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T09:56:15.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:15.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T09:56:15.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:15.001+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:15.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:15.001+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:15.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:15.001+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:19.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:20.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:20.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012a50 secure :-1 s=READY pgs=7865 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0abf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:20.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012a50 secure :-1 s=READY pgs=7865 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0abf70).stop 2021-03-09T09:56:20.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:20.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012a50 unknown :-1 s=CLOSED pgs=7865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:20.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:20.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0124c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:20.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:20.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ce20 con 0x7f21cc005e30 2021-03-09T09:56:20.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1c0 con 0x7f21c4003cb0 2021-03-09T09:56:20.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.993+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056a30 con 0x7f21cc02f1a0 2021-03-09T09:56:20.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:20.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:20.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:20.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:20.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0124c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:20.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:20.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:20.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:20.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:20.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086020 con 0x7f21cc02f1a0 2021-03-09T09:56:20.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04da10 con 0x7f21cc02f1a0 2021-03-09T09:56:20.007 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:19.997+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26627 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:20.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.001+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0124c0 secure :-1 s=READY pgs=7877 cs=0 l=1 rev1=1 rx=0x7f21e40a4f90 tx=0x7f21e4017190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:20.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009a10 con 0x7f21cc02f1a0 2021-03-09T09:56:20.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1b50 con 0x7f21cc02f1a0 2021-03-09T09:56:20.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1da0 con 0x7f21cc02f1a0 2021-03-09T09:56:20.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.005+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:20.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.005+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:20.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:20.005+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:23.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:23.117 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:25.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0124c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:56:25.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0124c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:25.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0124c0 secure :-1 s=READY pgs=7877 cs=0 l=1 rev1=1 rx=0x7f21e40a4f90 tx=0x7f21e4017190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:25.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0124c0 secure :-1 s=READY pgs=7877 cs=0 l=1 rev1=1 rx=0x7f21e40a4f90 tx=0x7f21e4017190).stop 2021-03-09T09:56:25.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0124c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:25.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0124c0 unknown :-1 s=CLOSED pgs=7877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:25.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:25.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:25.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:25.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093d40 con 0x7f21cc005e30 2021-03-09T09:56:25.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056a30 con 0x7f21c4003cb0 2021-03-09T09:56:25.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1c0 con 0x7f21c4008860 2021-03-09T09:56:25.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:25.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:25.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:25.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:25.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:25.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:25.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:25.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:25.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:25.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086020 con 0x7f21c4008860 2021-03-09T09:56:25.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.001+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:56:25.011 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:25.005+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26665 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:25.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.005+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 secure :-1 s=READY pgs=7889 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:25.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b34c0 con 0x7f21c4008860 2021-03-09T09:56:25.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21c4008860 2021-03-09T09:56:25.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T09:56:25.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.013+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:25.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.013+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:25.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:25.013+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:30.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:30.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:30.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 secure :-1 s=READY pgs=7889 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:30.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 secure :-1 s=READY pgs=7889 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009880).stop 2021-03-09T09:56:30.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:30.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 unknown :-1 s=CLOSED pgs=7889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:30.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:30.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:30.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:30.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ee90 con 0x7f21cc005e30 2021-03-09T09:56:30.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046b10 con 0x7f21c4003cb0 2021-03-09T09:56:30.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f740 con 0x7f21cc02f1a0 2021-03-09T09:56:30.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:30.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:30.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:30.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:30.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.004+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:30.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:30.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:30.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:30.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:30.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086020 con 0x7f21cc02f1a0 2021-03-09T09:56:30.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:56:30.016 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26701 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:30.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012180 secure :-1 s=READY pgs=7899 cs=0 l=1 rev1=1 rx=0x7f21dc062490 tx=0x7f21dc056190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:30.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc019d70 con 0x7f21cc02f1a0 2021-03-09T09:56:30.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc02f1a0 2021-03-09T09:56:30.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc02f1a0 2021-03-09T09:56:30.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.016+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:30.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.016+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:30.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:30.016+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:35.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:35.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:35.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012180 secure :-1 s=READY pgs=7899 cs=0 l=1 rev1=1 rx=0x7f21dc062490 tx=0x7f21dc056190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:35.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012180 secure :-1 s=READY pgs=7899 cs=0 l=1 rev1=1 rx=0x7f21dc062490 tx=0x7f21dc056190).stop 2021-03-09T09:56:35.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:35.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012180 unknown :-1 s=CLOSED pgs=7899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:35.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:35.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:35.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:35.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0532e0 con 0x7f21cc005e30 2021-03-09T09:56:35.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21c4003cb0 2021-03-09T09:56:35.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d460 con 0x7f21c4008860 2021-03-09T09:56:35.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:35.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:35.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:35.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:35.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:35.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:35.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:35.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:35.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:35.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f6f0 con 0x7f21c4008860 2021-03-09T09:56:35.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:56:35.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012670 secure :-1 s=READY pgs=7908 cs=0 l=1 rev1=1 rx=0x7f21dc02efd0 tx=0x7f21dc053040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:35.022 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:35.012+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26739 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:35.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099090 con 0x7f21c4008860 2021-03-09T09:56:35.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083380 con 0x7f21c4008860 2021-03-09T09:56:35.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02b230 con 0x7f21c4008860 2021-03-09T09:56:35.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.020+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:35.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.020+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:35.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:35.020+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:40.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:40.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.012+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:40.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012670 secure :-1 s=READY pgs=7908 cs=0 l=1 rev1=1 rx=0x7f21dc02efd0 tx=0x7f21dc053040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:40.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.012+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012670 secure :-1 s=READY pgs=7908 cs=0 l=1 rev1=1 rx=0x7f21dc02efd0 tx=0x7f21dc053040).stop 2021-03-09T09:56:40.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc012670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:40.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.012+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc012670 unknown :-1 s=CLOSED pgs=7908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:40.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0120f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:40.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:40.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:40.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21cc005e30 2021-03-09T09:56:40.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0532c0 con 0x7f21c4003cb0 2021-03-09T09:56:40.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:56:40.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:40.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:40.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:40.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:40.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0120f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:40.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:40.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:40.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:40.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:40.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012b60 con 0x7f21cc02f1a0 2021-03-09T09:56:40.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f6f0 con 0x7f21cc02f1a0 2021-03-09T09:56:40.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0120f0 secure :-1 s=READY pgs=7918 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc052e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:40.026 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:40.016+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26775 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:40.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21cc02f1a0 2021-03-09T09:56:40.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05bca0 con 0x7f21cc02f1a0 2021-03-09T09:56:40.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc02f1a0 2021-03-09T09:56:40.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.024+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:40.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.024+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:40.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:40.024+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:45.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0120f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0120f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0120f0 secure :-1 s=READY pgs=7918 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc052e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0120f0 secure :-1 s=READY pgs=7918 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc052e80).stop 2021-03-09T09:56:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0120f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0120f0 unknown :-1 s=CLOSED pgs=7918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0125e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21cc005e30 2021-03-09T09:56:45.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0532c0 con 0x7f21c4003cb0 2021-03-09T09:56:45.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f710 con 0x7f21c4008860 2021-03-09T09:56:45.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:45.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:45.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:45.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:45.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:45.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:45.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:45.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:45.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:45.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00efe0 con 0x7f21c4008860 2021-03-09T09:56:45.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b60 con 0x7f21c4008860 2021-03-09T09:56:45.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0125e0 secure :-1 s=READY pgs=7928 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc09b880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:45.030 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:45.024+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26813 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:45.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc099070 con 0x7f21c4008860 2021-03-09T09:56:45.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21c4008860 2021-03-09T09:56:45.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21c4008860 2021-03-09T09:56:45.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.028+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:45.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.028+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:45.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:45.028+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:50.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:50.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:50.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0125e0 secure :-1 s=READY pgs=7928 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc09b880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:50.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0125e0 secure :-1 s=READY pgs=7928 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc09b880).stop 2021-03-09T09:56:50.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0125e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:50.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0125e0 unknown :-1 s=CLOSED pgs=7928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:50.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:50.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21cc005e30 2021-03-09T09:56:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0532c0 con 0x7f21c4003cb0 2021-03-09T09:56:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:56:50.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:50.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:50.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f6f0 con 0x7f21cc02f1a0 2021-03-09T09:56:50.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00efe0 con 0x7f21cc02f1a0 2021-03-09T09:56:50.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.024+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011f00 secure :-1 s=READY pgs=7939 cs=0 l=1 rev1=1 rx=0x7f21dc009cf0 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:50.036 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:50.028+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26848 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:50.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02add0 con 0x7f21cc02f1a0 2021-03-09T09:56:50.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02aba0 con 0x7f21cc02f1a0 2021-03-09T09:56:50.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc02f1a0 2021-03-09T09:56:50.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.032+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:50.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:50.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:50.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:56:53.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:53.161 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:56:55.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc011f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:55.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc011f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:55.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011f00 secure :-1 s=READY pgs=7939 cs=0 l=1 rev1=1 rx=0x7f21dc009cf0 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:55.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011f00 secure :-1 s=READY pgs=7939 cs=0 l=1 rev1=1 rx=0x7f21dc009cf0 tx=0x7f21dc08a120).stop 2021-03-09T09:56:55.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc011f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:55.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011f00 unknown :-1 s=CLOSED pgs=7939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:55.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:55.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:55.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0123f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:55.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21cc005e30 2021-03-09T09:56:55.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0532c0 con 0x7f21c4003cb0 2021-03-09T09:56:55.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:56:55.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:56:55.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:55.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:56:55.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:55.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:55.038 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26887 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:56:55.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:55.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:55.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:55.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:55.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0128e0 con 0x7f21c4008860 2021-03-09T09:56:55.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f6f0 con 0x7f21c4008860 2021-03-09T09:56:55.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0123f0 secure :-1 s=READY pgs=7954 cs=0 l=1 rev1=1 rx=0x7f21dc0530b0 tx=0x7f21dc0bb2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:55.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02a9c0 con 0x7f21c4008860 2021-03-09T09:56:55.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T09:56:55.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21c4008860 2021-03-09T09:56:55.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.040+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:56:55.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:56:55.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:56:55.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:00.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0123f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:00.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0123f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:00.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0123f0 secure :-1 s=READY pgs=7954 cs=0 l=1 rev1=1 rx=0x7f21dc0530b0 tx=0x7f21dc0bb2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:00.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0123f0 secure :-1 s=READY pgs=7954 cs=0 l=1 rev1=1 rx=0x7f21dc0530b0 tx=0x7f21dc0bb2a0).stop 2021-03-09T09:57:00.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0123f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:00.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0123f0 unknown :-1 s=CLOSED pgs=7954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:00.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:00.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:00.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:00.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21cc005e30 2021-03-09T09:57:00.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21c4003cb0 2021-03-09T09:57:00.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:57:00.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:00.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:00.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:00.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:00.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.032+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:00.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:00.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:00.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:00.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:00.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:57:00.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0128e0 con 0x7f21cc02f1a0 2021-03-09T09:57:00.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011e60 secure :-1 s=READY pgs=7960 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc00c400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:00.047 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:00.036+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26922 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:00.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc098130 con 0x7f21cc02f1a0 2021-03-09T09:57:00.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6530 con 0x7f21cc02f1a0 2021-03-09T09:57:00.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21cc02f1a0 2021-03-09T09:57:00.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.044+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:00.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:00.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:00.044+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:05.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc011e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:05.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc011e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:05.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011e60 secure :-1 s=READY pgs=7960 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc00c400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:05.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011e60 secure :-1 s=READY pgs=7960 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc00c400).stop 2021-03-09T09:57:05.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc011e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:05.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc011e60 unknown :-1 s=CLOSED pgs=7960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:05.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:05.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:05.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:05.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21cc005e30 2021-03-09T09:57:05.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21c4003cb0 2021-03-09T09:57:05.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21c4008860 2021-03-09T09:57:05.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:05.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:05.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:05.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:05.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:05.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:05.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:05.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:05.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:05.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f6f0 con 0x7f21c4008860 2021-03-09T09:57:05.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.040+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:57:05.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.044+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 secure :-1 s=READY pgs=7971 cs=0 l=1 rev1=1 rx=0x7f21dc0184c0 tx=0x7f21dc098dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:05.053 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:05.044+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26961 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:05.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0329a0 con 0x7f21c4008860 2021-03-09T09:57:05.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6120 con 0x7f21c4008860 2021-03-09T09:57:05.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.048+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02ad90 con 0x7f21c4008860 2021-03-09T09:57:05.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.048+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:05.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.048+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:05.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:05.048+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:10.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:10.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:10.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 secure :-1 s=READY pgs=7971 cs=0 l=1 rev1=1 rx=0x7f21dc0184c0 tx=0x7f21dc098dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:10.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 secure :-1 s=READY pgs=7971 cs=0 l=1 rev1=1 rx=0x7f21dc0184c0 tx=0x7f21dc098dd0).stop 2021-03-09T09:57:10.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:10.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013ba0 unknown :-1 s=CLOSED pgs=7971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:10.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:10.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:10.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:10.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21cc005e30 2021-03-09T09:57:10.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21c4003cb0 2021-03-09T09:57:10.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dba0 con 0x7f21cc02f1a0 2021-03-09T09:57:10.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:10.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:10.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:10.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:10.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:10.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:10.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:10.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:10.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:10.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:57:10.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f6f0 con 0x7f21cc02f1a0 2021-03-09T09:57:10.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012000 secure :-1 s=READY pgs=7982 cs=0 l=1 rev1=1 rx=0x7f21dc06e050 tx=0x7f21dc018230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:10.059 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:10.048+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26995 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:10.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T09:57:10.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T09:57:10.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.056+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02ad90 con 0x7f21cc02f1a0 2021-03-09T09:57:10.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.056+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:10.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.060+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:10.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:10.060+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:15.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:15.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:15.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012000 secure :-1 s=READY pgs=7982 cs=0 l=1 rev1=1 rx=0x7f21dc06e050 tx=0x7f21dc018230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:15.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012000 secure :-1 s=READY pgs=7982 cs=0 l=1 rev1=1 rx=0x7f21dc06e050 tx=0x7f21dc018230).stop 2021-03-09T09:57:15.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc012000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:15.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc012000 unknown :-1 s=CLOSED pgs=7982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:15.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:15.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:15.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:15.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:15.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21c4003cb0 2021-03-09T09:57:15.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21c4008860 2021-03-09T09:57:15.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:15.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:15.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:15.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:15.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:15.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:15.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:15.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:15.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:15.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:57:15.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:57:15.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013b50 secure :-1 s=READY pgs=7992 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:15.068 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:15.056+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27035 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:15.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.056+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21c4008860 2021-03-09T09:57:15.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.056+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0781f0 con 0x7f21c4008860 2021-03-09T09:57:15.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.060+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02ad90 con 0x7f21c4008860 2021-03-09T09:57:15.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.060+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:15.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.060+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:15.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:15.060+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:20.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:20.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:20.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013b50 secure :-1 s=READY pgs=7992 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:20.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013b50 secure :-1 s=READY pgs=7992 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc009880).stop 2021-03-09T09:57:20.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc013b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:20.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc013b50 unknown :-1 s=CLOSED pgs=7992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:20.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc014040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:20.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:20.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:20.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:20.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091910 con 0x7f21c4003cb0 2021-03-09T09:57:20.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21cc02f1a0 2021-03-09T09:57:20.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:20.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:20.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:20.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:20.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc014040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:20.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:20.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:20.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:20.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:20.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:57:20.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:57:20.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc014040 secure :-1 s=READY pgs=8003 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc008ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:20.068 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:20.059+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27070 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:20.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc056500 con 0x7f21cc02f1a0 2021-03-09T09:57:20.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056320 con 0x7f21cc02f1a0 2021-03-09T09:57:20.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc02ad90 con 0x7f21cc02f1a0 2021-03-09T09:57:20.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.071+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:20.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.071+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:20.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:20.071+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:23.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:23.210 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:25.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc014040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:25.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc014040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:25.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc014040 secure :-1 s=READY pgs=8003 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc008ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:25.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc014040 secure :-1 s=READY pgs=8003 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc008ea0).stop 2021-03-09T09:57:25.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc014040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:25.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc014040 unknown :-1 s=CLOSED pgs=8003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:25.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:25.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:25.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc01d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:25.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:25.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21c4003cb0 2021-03-09T09:57:25.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e910 con 0x7f21c4008860 2021-03-09T09:57:25.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:25.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:25.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:25.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:25.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc01d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:25.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:25.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:25.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:25.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:25.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f6f0 con 0x7f21c4008860 2021-03-09T09:57:25.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:57:25.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc01d830 secure :-1 s=READY pgs=8014 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc056390).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:25.073 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27109 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:25.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bbed0 con 0x7f21c4008860 2021-03-09T09:57:25.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T09:57:25.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc085000 con 0x7f21c4008860 2021-03-09T09:57:25.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.071+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:25.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.071+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:25.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:25.071+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:30.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc01d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:57:30.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc01d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:30.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc01d830 secure :-1 s=READY pgs=8014 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc056390).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:30.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc01d830 secure :-1 s=READY pgs=8014 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc056390).stop 2021-03-09T09:57:30.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc01d830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:30.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc01d830 unknown :-1 s=CLOSED pgs=8014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:30.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:30.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:30.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:30.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:30.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21c4003cb0 2021-03-09T09:57:30.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21cc02f1a0 2021-03-09T09:57:30.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:30.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:30.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:30.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:30.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.067+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:30.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:30.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:30.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:30.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:30.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:57:30.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04da10 con 0x7f21cc02f1a0 2021-03-09T09:57:30.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015b60 secure :-1 s=READY pgs=8024 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4017350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:30.079 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27144 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:30.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc02f1a0 2021-03-09T09:57:30.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009390 con 0x7f21cc02f1a0 2021-03-09T09:57:30.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc02f1a0 2021-03-09T09:57:30.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.079+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:30.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.079+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:30.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:30.079+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:35.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:35.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:35.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015b60 secure :-1 s=READY pgs=8024 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4017350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:35.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015b60 secure :-1 s=READY pgs=8024 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4017350).stop 2021-03-09T09:57:35.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:35.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015b60 unknown :-1 s=CLOSED pgs=8024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:35.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.071+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:35.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:35.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:35.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:35.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21c4003cb0 2021-03-09T09:57:35.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027620 con 0x7f21c4008860 2021-03-09T09:57:35.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:35.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:35.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:35.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:35.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:35.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:35.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:35.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:35.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:35.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:57:35.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:57:35.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.075+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 secure :-1 s=READY pgs=8035 cs=0 l=1 rev1=1 rx=0x7f21e40b6000 tx=0x7f21e407f170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:35.083 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:35.079+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27183 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:35.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4090630 con 0x7f21c4008860 2021-03-09T09:57:35.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4008860 2021-03-09T09:57:35.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T09:57:35.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.083+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:35.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.087+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:35.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:35.087+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:40.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0846d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:40.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0846d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:40.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 secure :-1 s=READY pgs=8035 cs=0 l=1 rev1=1 rx=0x7f21e40b6000 tx=0x7f21e407f170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:40.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 secure :-1 s=READY pgs=8035 cs=0 l=1 rev1=1 rx=0x7f21e40b6000 tx=0x7f21e407f170).stop 2021-03-09T09:57:40.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0846d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:40.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 unknown :-1 s=CLOSED pgs=8035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:40.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:40.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:40.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:40.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:40.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e210 con 0x7f21c4003cb0 2021-03-09T09:57:40.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d530 con 0x7f21cc02f1a0 2021-03-09T09:57:40.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:40.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:40.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:40.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:40.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:40.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:40.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:40.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:40.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.079+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:40.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.083+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:57:40.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.083+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:57:40.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.083+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027290 secure :-1 s=READY pgs=8047 cs=0 l=1 rev1=1 rx=0x7f21e40097d0 tx=0x7f21e4033d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:40.088 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:40.083+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27218 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:40.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4090630 con 0x7f21cc02f1a0 2021-03-09T09:57:40.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc02f1a0 2021-03-09T09:57:40.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc02f1a0 2021-03-09T09:57:40.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.091+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:40.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.091+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:40.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:40.091+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:45.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:45.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:45.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027290 secure :-1 s=READY pgs=8047 cs=0 l=1 rev1=1 rx=0x7f21e40097d0 tx=0x7f21e4033d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:45.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027290 secure :-1 s=READY pgs=8047 cs=0 l=1 rev1=1 rx=0x7f21e40097d0 tx=0x7f21e4033d80).stop 2021-03-09T09:57:45.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:45.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027290 unknown :-1 s=CLOSED pgs=8047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:45.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:45.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:45.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:45.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e210 con 0x7f21cc005e30 2021-03-09T09:57:45.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03b590 con 0x7f21c4003cb0 2021-03-09T09:57:45.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21c4008860 2021-03-09T09:57:45.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:45.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:45.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:45.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:45.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.083+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:45.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:45.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:45.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:45.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:45.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04da10 con 0x7f21c4008860 2021-03-09T09:57:45.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f6f0 con 0x7f21c4008860 2021-03-09T09:57:45.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 secure :-1 s=READY pgs=8057 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:45.094 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:45.087+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27257 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:45.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21c4008860 2021-03-09T09:57:45.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6120 con 0x7f21c4008860 2021-03-09T09:57:45.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06ac90 con 0x7f21c4008860 2021-03-09T09:57:45.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.095+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:45.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.095+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:45.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:45.095+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:50.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0846d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:50.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0846d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:50.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 secure :-1 s=READY pgs=8057 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:50.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 secure :-1 s=READY pgs=8057 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc009810).stop 2021-03-09T09:57:50.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0846d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:50.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0846d0 unknown :-1 s=CLOSED pgs=8057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:50.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:50.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:50.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:50.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e210 con 0x7f21cc005e30 2021-03-09T09:57:50.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e920 con 0x7f21c4003cb0 2021-03-09T09:57:50.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004670 con 0x7f21cc02f1a0 2021-03-09T09:57:50.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:50.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:50.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:50.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:50.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:50.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:50.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:50.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:50.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:50.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:57:50.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:57:50.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.091+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015aa0 secure :-1 s=READY pgs=8067 cs=0 l=1 rev1=1 rx=0x7f21e4017320 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:50.107 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:50.095+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27292 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:50.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40052e0 con 0x7f21cc02f1a0 2021-03-09T09:57:50.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40054b0 con 0x7f21cc02f1a0 2021-03-09T09:57:50.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc02f1a0 2021-03-09T09:57:50.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.099+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:50.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.099+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:50.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:50.099+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:57:53.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:53.258 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:57:55.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:55.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:55.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015aa0 secure :-1 s=READY pgs=8067 cs=0 l=1 rev1=1 rx=0x7f21e4017320 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:55.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015aa0 secure :-1 s=READY pgs=8067 cs=0 l=1 rev1=1 rx=0x7f21e4017320 tx=0x7f21e4053ed0).stop 2021-03-09T09:57:55.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:55.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015aa0 unknown :-1 s=CLOSED pgs=8067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:55.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:55.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:55.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:55.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028070 con 0x7f21cc005e30 2021-03-09T09:57:55.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004670 con 0x7f21c4003cb0 2021-03-09T09:57:55.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012520 con 0x7f21c4008860 2021-03-09T09:57:55.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:57:55.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:55.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:57:55.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:55.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.095+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:55.102 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27331 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:57:55.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:55.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:55.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:55.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:55.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:57:55.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:57:55.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 secure :-1 s=READY pgs=8082 cs=0 l=1 rev1=1 rx=0x7f21cc056490 tx=0x7f21e408a290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:55.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.103+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21c4008860 2021-03-09T09:57:55.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.107+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21c4008860 2021-03-09T09:57:55.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.107+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041c30 con 0x7f21c4008860 2021-03-09T09:57:55.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.107+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:57:55.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.107+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:57:55.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:57:55.107+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:00.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc00f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:00.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc00f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:00.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 secure :-1 s=READY pgs=8082 cs=0 l=1 rev1=1 rx=0x7f21cc056490 tx=0x7f21e408a290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:00.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 secure :-1 s=READY pgs=8082 cs=0 l=1 rev1=1 rx=0x7f21cc056490 tx=0x7f21e408a290).stop 2021-03-09T09:58:00.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc00f510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:00.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 unknown :-1 s=CLOSED pgs=8082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:00.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:00.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:00.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:00.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004670 con 0x7f21cc005e30 2021-03-09T09:58:00.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053460 con 0x7f21c4003cb0 2021-03-09T09:58:00.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012500 con 0x7f21cc02f1a0 2021-03-09T09:58:00.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:00.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:00.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:00.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:00.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:00.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:00.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:00.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.099+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:00.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.103+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:00.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.103+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03fd10 con 0x7f21cc02f1a0 2021-03-09T09:58:00.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.103+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:58:00.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.103+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8088 cs=0 l=1 rev1=1 rx=0x7f21e401dcc0 tx=0x7f21e4038ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:00.108 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:00.103+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27366 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:00.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.107+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050090 con 0x7f21cc02f1a0 2021-03-09T09:58:00.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.107+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc02f1a0 2021-03-09T09:58:00.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.107+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc02f1a0 2021-03-09T09:58:00.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.107+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:00.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.111+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:00.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:00.111+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:05.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:05.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:05.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8088 cs=0 l=1 rev1=1 rx=0x7f21e401dcc0 tx=0x7f21e4038ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:05.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8088 cs=0 l=1 rev1=1 rx=0x7f21e401dcc0 tx=0x7f21e4038ca0).stop 2021-03-09T09:58:05.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:05.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:05.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:05.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:05.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:05.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004670 con 0x7f21cc005e30 2021-03-09T09:58:05.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012500 con 0x7f21c4003cb0 2021-03-09T09:58:05.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:58:05.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:05.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:05.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:05.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.103+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:05.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:05.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:05.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:05.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:05.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:05.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:58:05.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04da10 con 0x7f21c4008860 2021-03-09T09:58:05.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 secure :-1 s=READY pgs=8100 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:05.117 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:05.107+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27405 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:05.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.111+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4008860 2021-03-09T09:58:05.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.111+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4008860 2021-03-09T09:58:05.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21c4008860 2021-03-09T09:58:05.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.115+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:05.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.115+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:05.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:05.115+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:10.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.107+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc00f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:58:10.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc00f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:10.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 secure :-1 s=READY pgs=8100 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:10.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 secure :-1 s=READY pgs=8100 cs=0 l=1 rev1=1 rx=0x7f21dc065010 tx=0x7f21dc010d20).stop 2021-03-09T09:58:10.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc00f510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:10.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc00f510 unknown :-1 s=CLOSED pgs=8100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:10.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:10.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:10.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:10.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004670 con 0x7f21cc005e30 2021-03-09T09:58:10.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012500 con 0x7f21c4003cb0 2021-03-09T09:58:10.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:10.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:58:10.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:10.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:10.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:10.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:10.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:10.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:10.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:10.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:10.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04da10 con 0x7f21cc02f1a0 2021-03-09T09:58:10.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:58:10.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.111+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0baec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:10.119 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:10.115+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27440 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:10.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc02f1a0 2021-03-09T09:58:10.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.115+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ba9b0 con 0x7f21cc02f1a0 2021-03-09T09:58:10.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.119+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc02f1a0 2021-03-09T09:58:10.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.119+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:10.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.119+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:10.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:10.119+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:15.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:15.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:15.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0baec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:15.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0baec0).stop 2021-03-09T09:58:15.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:15.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:15.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:15.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:15.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:15.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094f10 con 0x7f21cc005e30 2021-03-09T09:58:15.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ea0 con 0x7f21c4003cb0 2021-03-09T09:58:15.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041120 con 0x7f21c4008860 2021-03-09T09:58:15.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:15.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:15.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:15.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:15.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:15.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:15.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:15.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:15.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:15.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012500 con 0x7f21c4008860 2021-03-09T09:58:15.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04da10 con 0x7f21c4008860 2021-03-09T09:58:15.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc007040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:15.124 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:15.118+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27479 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:15.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21c4008860 2021-03-09T09:58:15.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ba9b0 con 0x7f21c4008860 2021-03-09T09:58:15.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21c4008860 2021-03-09T09:58:15.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.126+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:15.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.126+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:15.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:15.126+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:20.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:20.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:20.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc007040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:20.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc007040).stop 2021-03-09T09:58:20.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:20.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=CLOSED pgs=8120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:20.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:20.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:20.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:20.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004670 con 0x7f21cc005e30 2021-03-09T09:58:20.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009c00 con 0x7f21c4003cb0 2021-03-09T09:58:20.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04da10 con 0x7f21cc02f1a0 2021-03-09T09:58:20.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:20.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:20.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:20.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:20.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:20.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:20.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:20.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:20.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:20.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:58:20.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012500 con 0x7f21cc02f1a0 2021-03-09T09:58:20.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8132 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:20.130 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:20.122+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27514 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:20.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc02f1a0 2021-03-09T09:58:20.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6ce0 con 0x7f21cc02f1a0 2021-03-09T09:58:20.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc09a070 con 0x7f21cc02f1a0 2021-03-09T09:58:20.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.130+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:20.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.130+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:20.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:20.130+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:23.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:23.346 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:25.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:25.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:25.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8132 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:25.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8132 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6f70).stop 2021-03-09T09:58:25.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:25.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:25.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:25.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:25.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:25.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04da10 con 0x7f21cc005e30 2021-03-09T09:58:25.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21c4003cb0 2021-03-09T09:58:25.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011e60 con 0x7f21c4008860 2021-03-09T09:58:25.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:25.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:25.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:25.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:25.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:25.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:25.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:25.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:25.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:25.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:58:25.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:58:25.134 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:25.130+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:25.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.130+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8146 cs=0 l=1 rev1=1 rx=0x7f21dc07ad50 tx=0x7f21dc018150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:25.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02bf00 con 0x7f21c4008860 2021-03-09T09:58:25.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc02bf00 con 0x7f21c4008860 2021-03-09T09:58:25.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc09a070 con 0x7f21c4008860 2021-03-09T09:58:25.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.138+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:25.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.138+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:25.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:25.138+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:30.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:30.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:30.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8146 cs=0 l=1 rev1=1 rx=0x7f21dc07ad50 tx=0x7f21dc018150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:30.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8146 cs=0 l=1 rev1=1 rx=0x7f21dc07ad50 tx=0x7f21dc018150).stop 2021-03-09T09:58:30.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:30.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=CLOSED pgs=8146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:30.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:30.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:30.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:30.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:30.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053460 con 0x7f21cc005e30 2021-03-09T09:58:30.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:30.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04da10 con 0x7f21c4003cb0 2021-03-09T09:58:30.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21cc02f1a0 2021-03-09T09:58:30.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:30.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:30.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:30.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:30.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:30.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:30.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:30.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040ee0 con 0x7f21cc02f1a0 2021-03-09T09:58:30.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03fd10 con 0x7f21cc02f1a0 2021-03-09T09:58:30.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.134+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8152 cs=0 l=1 rev1=1 rx=0x7f21cc012a60 tx=0x7f21e407f170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:30.141 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:30.134+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27588 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:30.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050070 con 0x7f21cc02f1a0 2021-03-09T09:58:30.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4008530 con 0x7f21cc02f1a0 2021-03-09T09:58:30.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21cc02f1a0 2021-03-09T09:58:30.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.138+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:30.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.138+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:30.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:30.138+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:35.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:35.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:35.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8152 cs=0 l=1 rev1=1 rx=0x7f21cc012a60 tx=0x7f21e407f170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:35.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8152 cs=0 l=1 rev1=1 rx=0x7f21cc012a60 tx=0x7f21e407f170).stop 2021-03-09T09:58:35.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:35.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:35.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:35.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:35.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:35.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023d70 con 0x7f21cc005e30 2021-03-09T09:58:35.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21c4003cb0 2021-03-09T09:58:35.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:58:35.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:35.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:35.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:35.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:35.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.134+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:35.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:35.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:35.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:35.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:35.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011e60 con 0x7f21c4008860 2021-03-09T09:58:35.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040ee0 con 0x7f21c4008860 2021-03-09T09:58:35.144 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27627 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:35.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 rx=0x7f21e4048f80 tx=0x7f21e4009a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:35.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050070 con 0x7f21c4008860 2021-03-09T09:58:35.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21c4008860 2021-03-09T09:58:35.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c4008860 2021-03-09T09:58:35.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.146+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:35.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.146+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:35.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:35.146+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:40.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:40.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:40.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 rx=0x7f21e4048f80 tx=0x7f21e4009a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:40.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 rx=0x7f21e4048f80 tx=0x7f21e4009a10).stop 2021-03-09T09:58:40.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:40.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=CLOSED pgs=8163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:40.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:40.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:40.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:40.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023d70 con 0x7f21cc005e30 2021-03-09T09:58:40.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040070 con 0x7f21c4003cb0 2021-03-09T09:58:40.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21cc02f1a0 2021-03-09T09:58:40.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:40.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:40.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:40.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:40.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.138+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:40.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:40.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:40.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:40.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:40.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:58:40.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:58:40.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8174 cs=0 l=1 rev1=1 rx=0x7f21dc0bb570 tx=0x7f21dc09bea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:40.149 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:40.142+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27662 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:40.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc009810 con 0x7f21cc02f1a0 2021-03-09T09:58:40.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06a5b0 con 0x7f21cc02f1a0 2021-03-09T09:58:40.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc02f1a0 2021-03-09T09:58:40.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.146+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:40.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.150+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:40.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:40.150+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:45.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:58:45.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:45.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8174 cs=0 l=1 rev1=1 rx=0x7f21dc0bb570 tx=0x7f21dc09bea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:45.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8174 cs=0 l=1 rev1=1 rx=0x7f21dc0bb570 tx=0x7f21dc09bea0).stop 2021-03-09T09:58:45.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:45.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:45.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:45.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:45.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:45.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040070 con 0x7f21cc005e30 2021-03-09T09:58:45.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ce00 con 0x7f21c4003cb0 2021-03-09T09:58:45.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023d70 con 0x7f21c4008860 2021-03-09T09:58:45.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:45.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:45.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:45.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:45.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:45.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:45.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:45.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:45.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:45.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:58:45.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011e60 con 0x7f21c4008860 2021-03-09T09:58:45.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8184 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4033790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:45.154 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:45.150+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27700 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:45.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21c4008860 2021-03-09T09:58:45.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21c4008860 2021-03-09T09:58:45.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T09:58:45.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.154+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:45.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:45.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:45.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8184 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4033790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8184 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4033790).stop 2021-03-09T09:58:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=CLOSED pgs=8184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:50.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:50.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:50.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ce00 con 0x7f21cc005e30 2021-03-09T09:58:50.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040070 con 0x7f21c4003cb0 2021-03-09T09:58:50.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023d70 con 0x7f21cc02f1a0 2021-03-09T09:58:50.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:50.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:50.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:50.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:50.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.150+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:50.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:50.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:50.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:50.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:50.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017990 con 0x7f21cc02f1a0 2021-03-09T09:58:50.159 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27736 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:50.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:58:50.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8199 cs=0 l=1 rev1=1 rx=0x7f21dc019bf0 tx=0x7f21dc0ab140).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:50.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0576e0 con 0x7f21cc02f1a0 2021-03-09T09:58:50.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c280 con 0x7f21cc02f1a0 2021-03-09T09:58:50.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc02f1a0 2021-03-09T09:58:50.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.162+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:50.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.162+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:50.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:50.162+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:58:53.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:53.425 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:58:55.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:55.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:55.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8199 cs=0 l=1 rev1=1 rx=0x7f21dc019bf0 tx=0x7f21dc0ab140).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:55.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8199 cs=0 l=1 rev1=1 rx=0x7f21dc019bf0 tx=0x7f21dc0ab140).stop 2021-03-09T09:58:55.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:55.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:55.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:55.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:55.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:55.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0478e0 con 0x7f21cc005e30 2021-03-09T09:58:55.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023d70 con 0x7f21c4003cb0 2021-03-09T09:58:55.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21c4008860 2021-03-09T09:58:55.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:58:55.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:58:55.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:55.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:55.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:55.163 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27774 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:58:55.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053460 con 0x7f21c4008860 2021-03-09T09:58:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017990 con 0x7f21c4008860 2021-03-09T09:58:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8212 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc00c480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:55.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.170+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06a280 con 0x7f21c4008860 2021-03-09T09:58:55.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.170+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ac1e0 con 0x7f21c4008860 2021-03-09T09:58:55.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.170+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06a070 con 0x7f21c4008860 2021-03-09T09:58:55.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.170+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:58:55.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.170+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:58:55.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:58:55.170+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:00.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:00.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:00.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8212 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc00c480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:00.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 secure :-1 s=READY pgs=8212 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc00c480).stop 2021-03-09T09:59:00.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0159c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:00.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0159c0 unknown :-1 s=CLOSED pgs=8212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:00.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:00.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:00.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:00.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ce00 con 0x7f21cc005e30 2021-03-09T09:59:00.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0955a0 con 0x7f21c4003cb0 2021-03-09T09:59:00.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:00.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0126e0 con 0x7f21cc02f1a0 2021-03-09T09:59:00.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:00.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:00.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:00.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:00.169 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:00.166+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27811 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:00.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:00.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:00.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:00.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:00.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017b30 con 0x7f21cc02f1a0 2021-03-09T09:59:00.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.162+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:59:00.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.166+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8217 cs=0 l=1 rev1=1 rx=0x7f21e40a5290 tx=0x7f21e4009af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:00.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.170+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050020 con 0x7f21cc02f1a0 2021-03-09T09:59:00.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.170+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4034650 con 0x7f21cc02f1a0 2021-03-09T09:59:00.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.174+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4034000 con 0x7f21cc02f1a0 2021-03-09T09:59:00.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.174+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:00.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.174+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:00.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:00.174+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:05.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:05.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:05.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8217 cs=0 l=1 rev1=1 rx=0x7f21e40a5290 tx=0x7f21e4009af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:05.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 secure :-1 s=READY pgs=8217 cs=0 l=1 rev1=1 rx=0x7f21e40a5290 tx=0x7f21e4009af0).stop 2021-03-09T09:59:05.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc027240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:05.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc027240 unknown :-1 s=CLOSED pgs=8217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:05.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:05.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc04a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:05.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:05.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ce00 con 0x7f21cc005e30 2021-03-09T09:59:05.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004a50 con 0x7f21c4003cb0 2021-03-09T09:59:05.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21c4008860 2021-03-09T09:59:05.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:05.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:05.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:05.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:05.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc04a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:05.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:05.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:05.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:05.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:05.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:59:05.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017b30 con 0x7f21c4008860 2021-03-09T09:59:05.175 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27848 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:05.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc04a890 secure :-1 s=READY pgs=8229 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e407fd60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:05.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21c4008860 2021-03-09T09:59:05.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4008860 2021-03-09T09:59:05.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f470 con 0x7f21c4008860 2021-03-09T09:59:05.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.177+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:05.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.177+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:05.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:05.177+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:10.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc04a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:10.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc04a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:10.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc04a890 secure :-1 s=READY pgs=8229 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e407fd60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:10.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc04a890 secure :-1 s=READY pgs=8229 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e407fd60).stop 2021-03-09T09:59:10.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc04a890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:10.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc04a890 unknown :-1 s=CLOSED pgs=8229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:10.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:10.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:10.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:10.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132d0 con 0x7f21cc005e30 2021-03-09T09:59:10.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004a30 con 0x7f21c4003cb0 2021-03-09T09:59:10.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T09:59:10.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:10.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:10.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:10.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:10.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:10.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:10.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:10.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:10.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:10.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027240 con 0x7f21cc02f1a0 2021-03-09T09:59:10.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053460 con 0x7f21cc02f1a0 2021-03-09T09:59:10.180 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27885 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:10.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8240 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:10.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00dec0 con 0x7f21cc02f1a0 2021-03-09T09:59:10.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00dec0 con 0x7f21cc02f1a0 2021-03-09T09:59:10.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc032220 con 0x7f21cc02f1a0 2021-03-09T09:59:10.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.181+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:10.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:10.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:10.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:15.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:15.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:15.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8240 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:15.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8240 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc009810).stop 2021-03-09T09:59:15.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:15.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:15.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:15.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:15.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0273d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:15.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0132f0 con 0x7f21cc005e30 2021-03-09T09:59:15.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21c4003cb0 2021-03-09T09:59:15.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011e60 con 0x7f21c4008860 2021-03-09T09:59:15.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:15.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:15.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:15.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:15.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:15.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:15.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:15.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:15.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:15.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a690 con 0x7f21c4008860 2021-03-09T09:59:15.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc027240 con 0x7f21c4008860 2021-03-09T09:59:15.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0273d0 secure :-1 s=READY pgs=8249 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc081070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:15.186 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:15.177+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27922 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:15.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21c4008860 2021-03-09T09:59:15.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc087250 con 0x7f21c4008860 2021-03-09T09:59:15.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T09:59:15.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.185+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:15.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:15.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:15.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:20.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:20.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:20.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0273d0 secure :-1 s=READY pgs=8249 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc081070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:20.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0273d0 secure :-1 s=READY pgs=8249 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc081070).stop 2021-03-09T09:59:20.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0273d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:20.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0273d0 unknown :-1 s=CLOSED pgs=8249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:20.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:20.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:20.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:20.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21cc005e30 2021-03-09T09:59:20.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0842d0 con 0x7f21c4003cb0 2021-03-09T09:59:20.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a820 con 0x7f21cc02f1a0 2021-03-09T09:59:20.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:20.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:20.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:20.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:20.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:20.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:20.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:20.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:20.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:20.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a960 con 0x7f21cc02f1a0 2021-03-09T09:59:20.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a690 con 0x7f21cc02f1a0 2021-03-09T09:59:20.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8259 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc00dec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:20.192 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:20.185+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27959 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:20.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0040e0 con 0x7f21cc02f1a0 2021-03-09T09:59:20.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0040e0 con 0x7f21cc02f1a0 2021-03-09T09:59:20.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T09:59:20.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.189+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:20.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:20.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:20.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:23.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:23.473 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:25.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:25.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:25.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8259 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc00dec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:25.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8259 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc00dec0).stop 2021-03-09T09:59:25.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:25.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:25.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:25.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:25.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc027300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:25.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21cc005e30 2021-03-09T09:59:25.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0842d0 con 0x7f21c4003cb0 2021-03-09T09:59:25.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ac90 con 0x7f21c4008860 2021-03-09T09:59:25.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:25.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:25.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc027300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:25.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:25.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:25.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:25.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.185+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:25.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:25.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:25.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21c4008860 2021-03-09T09:59:25.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a960 con 0x7f21c4008860 2021-03-09T09:59:25.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc027300 secure :-1 s=READY pgs=8271 cs=0 l=1 rev1=1 rx=0x7f21dc020600 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:25.196 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:25.189+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27995 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:25.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052e10 con 0x7f21c4008860 2021-03-09T09:59:25.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052e10 con 0x7f21c4008860 2021-03-09T09:59:25.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T09:59:25.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.201+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:25.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:25.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:25.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:30.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc027300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:30.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc027300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:30.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc027300 secure :-1 s=READY pgs=8271 cs=0 l=1 rev1=1 rx=0x7f21dc020600 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:30.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc027300 secure :-1 s=READY pgs=8271 cs=0 l=1 rev1=1 rx=0x7f21dc020600 tx=0x7f21dc009880).stop 2021-03-09T09:59:30.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc027300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:30.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc027300 unknown :-1 s=CLOSED pgs=8271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:30.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:30.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:30.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:30.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b4a0 con 0x7f21cc005e30 2021-03-09T09:59:30.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4003cb0 2021-03-09T09:59:30.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21cc02f1a0 2021-03-09T09:59:30.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:30.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:30.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:30.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:30.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:30.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:30.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:30.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:30.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:30.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084280 con 0x7f21cc02f1a0 2021-03-09T09:59:30.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T09:59:30.201 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:30.193+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28032 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:30.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8284 cs=0 l=1 rev1=1 rx=0x7f21dc0b6290 tx=0x7f21dc052d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:30.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc091070 con 0x7f21cc02f1a0 2021-03-09T09:59:30.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc02f1a0 2021-03-09T09:59:30.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ab780 con 0x7f21cc02f1a0 2021-03-09T09:59:30.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.201+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:30.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:30.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:30.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:35.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:35.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:35.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8284 cs=0 l=1 rev1=1 rx=0x7f21dc0b6290 tx=0x7f21dc052d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:35.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8284 cs=0 l=1 rev1=1 rx=0x7f21dc0b6290 tx=0x7f21dc052d00).stop 2021-03-09T09:59:35.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:35.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:35.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:35.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:35.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:35.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b4a0 con 0x7f21cc005e30 2021-03-09T09:59:35.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0160a0 con 0x7f21c4003cb0 2021-03-09T09:59:35.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026fc0 con 0x7f21c4008860 2021-03-09T09:59:35.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:35.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:35.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:35.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:35.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:35.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:35.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:35.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:35.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:35.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049bb0 con 0x7f21c4008860 2021-03-09T09:59:35.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.197+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc082db0 con 0x7f21c4008860 2021-03-09T09:59:35.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052f80 secure :-1 s=READY pgs=8293 cs=0 l=1 rev1=1 rx=0x7f21e404aff0 tx=0x7f21e4031440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:35.207 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:35.201+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28069 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:35.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c4008860 2021-03-09T09:59:35.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009800 con 0x7f21c4008860 2021-03-09T09:59:35.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T09:59:35.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.209+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:35.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.209+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:35.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:35.209+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:40.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:40.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:40.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052f80 secure :-1 s=READY pgs=8293 cs=0 l=1 rev1=1 rx=0x7f21e404aff0 tx=0x7f21e4031440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:40.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052f80 secure :-1 s=READY pgs=8293 cs=0 l=1 rev1=1 rx=0x7f21e404aff0 tx=0x7f21e4031440).stop 2021-03-09T09:59:40.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:40.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052f80 unknown :-1 s=CLOSED pgs=8293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:40.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:40.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:40.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:40.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0160a0 con 0x7f21cc005e30 2021-03-09T09:59:40.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f190 con 0x7f21c4003cb0 2021-03-09T09:59:40.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21cc02f1a0 2021-03-09T09:59:40.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:40.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:40.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:40.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:40.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:40.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:40.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:40.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:40.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:40.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007cc0 con 0x7f21cc02f1a0 2021-03-09T09:59:40.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.201+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084280 con 0x7f21cc02f1a0 2021-03-09T09:59:40.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.205+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8302 cs=0 l=1 rev1=1 rx=0x7f21dc092040 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:40.212 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:40.205+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28106 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:40.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.205+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081070 con 0x7f21cc02f1a0 2021-03-09T09:59:40.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.205+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3f10 con 0x7f21cc02f1a0 2021-03-09T09:59:40.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052dd0 con 0x7f21cc02f1a0 2021-03-09T09:59:40.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.213+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:40.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.213+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:40.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:40.213+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:45.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:45.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:45.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8302 cs=0 l=1 rev1=1 rx=0x7f21dc092040 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:45.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8302 cs=0 l=1 rev1=1 rx=0x7f21dc092040 tx=0x7f21dc08a120).stop 2021-03-09T09:59:45.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:45.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:45.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:45.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:45.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0918c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:45.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027660 con 0x7f21cc005e30 2021-03-09T09:59:45.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c4003cb0 2021-03-09T09:59:45.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:45.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:45.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052f10 con 0x7f21c4008860 2021-03-09T09:59:45.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:45.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:45.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0918c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:45.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:45.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:45.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:45.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:45.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21c4008860 2021-03-09T09:59:45.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007cc0 con 0x7f21c4008860 2021-03-09T09:59:45.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0918c0 secure :-1 s=READY pgs=8312 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0bb320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:45.217 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28143 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:45.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bbed0 con 0x7f21c4008860 2021-03-09T09:59:45.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0bbed0 con 0x7f21c4008860 2021-03-09T09:59:45.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc078ac0 con 0x7f21c4008860 2021-03-09T09:59:45.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.217+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:45.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.217+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:45.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:45.217+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:50.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0918c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:50.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0918c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:50.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0918c0 secure :-1 s=READY pgs=8312 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0bb320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:50.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0918c0 secure :-1 s=READY pgs=8312 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0bb320).stop 2021-03-09T09:59:50.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0918c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:50.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0918c0 unknown :-1 s=CLOSED pgs=8312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:50.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082dd0 con 0x7f21cc005e30 2021-03-09T09:59:50.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4003cb0 2021-03-09T09:59:50.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21cc02f1a0 2021-03-09T09:59:50.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:50.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:50.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:50.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:50.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:50.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:50.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:50.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:50.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:50.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052ff0 con 0x7f21cc02f1a0 2021-03-09T09:59:50.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049bb0 con 0x7f21cc02f1a0 2021-03-09T09:59:50.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.213+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8324 cs=0 l=1 rev1=1 rx=0x7f21e4080050 tx=0x7f21e4027eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:50.223 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:50.217+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28180 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:50.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050030 con 0x7f21cc02f1a0 2021-03-09T09:59:50.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21cc02f1a0 2021-03-09T09:59:50.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc02f1a0 2021-03-09T09:59:50.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.225+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:50.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.225+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:50.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:50.225+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T09:59:53.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:53.520 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T09:59:55.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:55.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:55.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8324 cs=0 l=1 rev1=1 rx=0x7f21e4080050 tx=0x7f21e4027eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:55.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8324 cs=0 l=1 rev1=1 rx=0x7f21e4080050 tx=0x7f21e4027eb0).stop 2021-03-09T09:59:55.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:55.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:55.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:55.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:55.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc091580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:55.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21cc005e30 2021-03-09T09:59:55.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082dd0 con 0x7f21c4003cb0 2021-03-09T09:59:55.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21c4008860 2021-03-09T09:59:55.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T09:59:55.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:55.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc091580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:55.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T09:59:55.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:55.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:55.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:55.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:55.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:55.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044b10 con 0x7f21c4008860 2021-03-09T09:59:55.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21c4008860 2021-03-09T09:59:55.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc091580 secure :-1 s=READY pgs=8334 cs=0 l=1 rev1=1 rx=0x7f21dc0a2040 tx=0x7f21dc015a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:55.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bbed0 con 0x7f21c4008860 2021-03-09T09:59:55.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21c4008860 2021-03-09T09:59:55.229 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T09:59:55.220+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28217 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T09:59:55.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.224+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21c4008860 2021-03-09T09:59:55.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.224+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T09:59:55.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.224+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T09:59:55.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T09:59:55.224+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:00.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc091580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:00.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc091580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:00.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc091580 secure :-1 s=READY pgs=8334 cs=0 l=1 rev1=1 rx=0x7f21dc0a2040 tx=0x7f21dc015a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:00.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc091580 secure :-1 s=READY pgs=8334 cs=0 l=1 rev1=1 rx=0x7f21dc0a2040 tx=0x7f21dc015a90).stop 2021-03-09T10:00:00.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc091580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:00.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc091580 unknown :-1 s=CLOSED pgs=8334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:00.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:00.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:00.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:00.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21cc005e30 2021-03-09T10:00:00.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21c4003cb0 2021-03-09T10:00:00.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T10:00:00.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:00.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:00.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:00.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:00.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:00.232 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28254 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:00.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:00.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:00.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:00.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:00.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007cc0 con 0x7f21cc02f1a0 2021-03-09T10:00:00.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.224+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044b10 con 0x7f21cc02f1a0 2021-03-09T10:00:00.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8349 cs=0 l=1 rev1=1 rx=0x7f21dc019690 tx=0x7f21dc0b8d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:00.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc02f1a0 2021-03-09T10:00:00.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056530 con 0x7f21cc02f1a0 2021-03-09T10:00:00.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc02f1a0 2021-03-09T10:00:00.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.232+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:00.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.236+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:00.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:00.236+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:05.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:05.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:05.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8349 cs=0 l=1 rev1=1 rx=0x7f21dc019690 tx=0x7f21dc0b8d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:05.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8349 cs=0 l=1 rev1=1 rx=0x7f21dc019690 tx=0x7f21dc0b8d20).stop 2021-03-09T10:00:05.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:05.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:05.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:05.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0914e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:05.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:05.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21cc005e30 2021-03-09T10:00:05.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21c4003cb0 2021-03-09T10:00:05.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21c4008860 2021-03-09T10:00:05.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:05.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:05.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:05.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:05.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0914e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:05.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:05.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:05.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:05.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:05.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0919d0 con 0x7f21c4008860 2021-03-09T10:00:05.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052ff0 con 0x7f21c4008860 2021-03-09T10:00:05.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.228+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0914e0 secure :-1 s=READY pgs=8355 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4055280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:05.239 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:05.232+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28291 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:05.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050070 con 0x7f21c4008860 2021-03-09T10:00:05.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T10:00:05.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T10:00:05.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.236+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:05.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.236+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:05.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:05.236+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:10.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0914e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:00:10.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0914e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:10.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0914e0 secure :-1 s=READY pgs=8355 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4055280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:10.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0914e0 secure :-1 s=READY pgs=8355 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4055280).stop 2021-03-09T10:00:10.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0914e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:10.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0914e0 unknown :-1 s=CLOSED pgs=8355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:10.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:10.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:10.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:10.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:10.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:10.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21cc005e30 2021-03-09T10:00:10.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21c4003cb0 2021-03-09T10:00:10.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T10:00:10.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:10.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:10.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.232+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:10.242 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28328 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:10.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:10.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:10.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:10.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:10.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091b60 con 0x7f21cc02f1a0 2021-03-09T10:00:10.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007cc0 con 0x7f21cc02f1a0 2021-03-09T10:00:10.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.236+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8371 cs=0 l=1 rev1=1 rx=0x7f21cc055800 tx=0x7f21dc015b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:10.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0196c0 con 0x7f21cc02f1a0 2021-03-09T10:00:10.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06a2a0 con 0x7f21cc02f1a0 2021-03-09T10:00:10.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T10:00:10.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.244+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:10.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.244+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:10.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:10.244+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:15.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:15.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:15.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8371 cs=0 l=1 rev1=1 rx=0x7f21cc055800 tx=0x7f21dc015b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:15.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8371 cs=0 l=1 rev1=1 rx=0x7f21cc055800 tx=0x7f21dc015b00).stop 2021-03-09T10:00:15.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:15.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:15.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:15.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:15.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:15.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a50 con 0x7f21cc005e30 2021-03-09T10:00:15.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21c4003cb0 2021-03-09T10:00:15.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ee0 con 0x7f21c4008860 2021-03-09T10:00:15.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:15.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:15.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:15.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:15.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:15.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:15.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:15.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:15.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:15.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049bb0 con 0x7f21c4008860 2021-03-09T10:00:15.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0919d0 con 0x7f21c4008860 2021-03-09T10:00:15.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8376 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e408f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:15.249 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:15.240+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28365 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:15.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050030 con 0x7f21c4008860 2021-03-09T10:00:15.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21c4008860 2021-03-09T10:00:15.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409ca40 con 0x7f21c4008860 2021-03-09T10:00:15.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.248+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:15.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.248+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:15.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:15.248+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:20.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:20.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:20.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8376 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e408f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:20.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8376 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e408f000).stop 2021-03-09T10:00:20.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:20.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=CLOSED pgs=8376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:20.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:20.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:20.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:20.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21cc005e30 2021-03-09T10:00:20.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0919d0 con 0x7f21c4003cb0 2021-03-09T10:00:20.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0478b0 con 0x7f21cc02f1a0 2021-03-09T10:00:20.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:20.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:20.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:20.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:20.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:20.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:20.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:20.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:20.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:20.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T10:00:20.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049bb0 con 0x7f21cc02f1a0 2021-03-09T10:00:20.254 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:20.248+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28401 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:20.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8389 cs=0 l=1 rev1=1 rx=0x7f21cc01d270 tx=0x7f21e40071e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:20.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409cf20 con 0x7f21cc02f1a0 2021-03-09T10:00:20.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409cf20 con 0x7f21cc02f1a0 2021-03-09T10:00:20.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409fca0 con 0x7f21cc02f1a0 2021-03-09T10:00:20.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.256+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:20.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.256+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:20.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:20.256+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:23.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:23.645 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:25.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:25.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:25.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8389 cs=0 l=1 rev1=1 rx=0x7f21cc01d270 tx=0x7f21e40071e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:25.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8389 cs=0 l=1 rev1=1 rx=0x7f21cc01d270 tx=0x7f21e40071e0).stop 2021-03-09T10:00:25.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:25.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:25.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:25.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:25.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:25.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21cc005e30 2021-03-09T10:00:25.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a30 con 0x7f21c4003cb0 2021-03-09T10:00:25.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056380 con 0x7f21c4008860 2021-03-09T10:00:25.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:25.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:25.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:25.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:25.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:25.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:25.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:25.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:25.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:25.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007cc0 con 0x7f21c4008860 2021-03-09T10:00:25.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.248+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091b60 con 0x7f21c4008860 2021-03-09T10:00:25.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.252+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8400 cs=0 l=1 rev1=1 rx=0x7f21dc018fe0 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:25.263 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:25.252+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28439 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:25.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T10:00:25.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00af80 con 0x7f21c4008860 2021-03-09T10:00:25.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T10:00:25.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.256+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:25.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.256+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:25.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:25.256+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:30.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:30.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:30.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8400 cs=0 l=1 rev1=1 rx=0x7f21dc018fe0 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:30.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8400 cs=0 l=1 rev1=1 rx=0x7f21dc018fe0 tx=0x7f21dc080000).stop 2021-03-09T10:00:30.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:30.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=CLOSED pgs=8400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:30.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:30.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:30.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:30.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21cc005e30 2021-03-09T10:00:30.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a30 con 0x7f21c4003cb0 2021-03-09T10:00:30.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21cc02f1a0 2021-03-09T10:00:30.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:30.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:30.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:30.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:30.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.252+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:30.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:30.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:30.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:30.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:30.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056330 con 0x7f21cc02f1a0 2021-03-09T10:00:30.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T10:00:30.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8408 cs=0 l=1 rev1=1 rx=0x7f21e4031560 tx=0x7f21e4016c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:30.265 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28472 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:30.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409cf20 con 0x7f21cc02f1a0 2021-03-09T10:00:30.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40363a0 con 0x7f21cc02f1a0 2021-03-09T10:00:30.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T10:00:30.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.260+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:30.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.264+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:30.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:30.264+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:35.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:35.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:35.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8408 cs=0 l=1 rev1=1 rx=0x7f21e4031560 tx=0x7f21e4016c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:35.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8408 cs=0 l=1 rev1=1 rx=0x7f21e4031560 tx=0x7f21e4016c10).stop 2021-03-09T10:00:35.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:35.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:35.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:35.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:35.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:35.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21cc005e30 2021-03-09T10:00:35.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a30 con 0x7f21c4003cb0 2021-03-09T10:00:35.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086490 con 0x7f21c4008860 2021-03-09T10:00:35.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:35.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:35.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:35.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:35.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:35.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:35.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:35.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:35.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:35.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049bb0 con 0x7f21c4008860 2021-03-09T10:00:35.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007cc0 con 0x7f21c4008860 2021-03-09T10:00:35.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8419 cs=0 l=1 rev1=1 rx=0x7f21dc0334b0 tx=0x7f21dc0bb4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:35.270 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:35.264+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:35.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bbed0 con 0x7f21c4008860 2021-03-09T10:00:35.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0bbed0 con 0x7f21c4008860 2021-03-09T10:00:35.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21c4008860 2021-03-09T10:00:35.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.268+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:35.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.272+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:35.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:35.272+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:40.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:40.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:40.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8419 cs=0 l=1 rev1=1 rx=0x7f21dc0334b0 tx=0x7f21dc0bb4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:40.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8419 cs=0 l=1 rev1=1 rx=0x7f21dc0334b0 tx=0x7f21dc0bb4e0).stop 2021-03-09T10:00:40.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:40.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=CLOSED pgs=8419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:40.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:40.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:40.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:40.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b780 con 0x7f21cc005e30 2021-03-09T10:00:40.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a30 con 0x7f21c4003cb0 2021-03-09T10:00:40.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041160 con 0x7f21cc02f1a0 2021-03-09T10:00:40.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:40.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:40.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:40.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:40.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:40.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:40.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:40.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:40.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:40.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21cc02f1a0 2021-03-09T10:00:40.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049bb0 con 0x7f21cc02f1a0 2021-03-09T10:00:40.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8430 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:40.277 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:40.268+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28549 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:40.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T10:00:40.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc02f1a0 2021-03-09T10:00:40.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc02f1a0 2021-03-09T10:00:40.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.272+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:40.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.276+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:40.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:40.276+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:45.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:45.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:45.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8430 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:45.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8430 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc009810).stop 2021-03-09T10:00:45.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:45.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:45.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:45.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:45.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:45.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:45.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a30 con 0x7f21cc005e30 2021-03-09T10:00:45.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:45.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bb0 con 0x7f21c4003cb0 2021-03-09T10:00:45.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4008860 2021-03-09T10:00:45.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:45.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:45.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:45.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:45.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:45.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:45.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:45.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0533a0 con 0x7f21c4008860 2021-03-09T10:00:45.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056330 con 0x7f21c4008860 2021-03-09T10:00:45.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8440 cs=0 l=1 rev1=1 rx=0x7f21e4036330 tx=0x7f21e408a290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:45.281 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:45.272+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28587 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:45.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024230 con 0x7f21c4008860 2021-03-09T10:00:45.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024230 con 0x7f21c4008860 2021-03-09T10:00:45.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cd10 con 0x7f21c4008860 2021-03-09T10:00:45.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.280+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:45.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.280+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:45.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:45.280+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:50.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8440 cs=0 l=1 rev1=1 rx=0x7f21e4036330 tx=0x7f21e408a290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 secure :-1 s=READY pgs=8440 cs=0 l=1 rev1=1 rx=0x7f21e4036330 tx=0x7f21e408a290).stop 2021-03-09T10:00:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:50.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052eb0 unknown :-1 s=CLOSED pgs=8440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:50.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:50.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:50.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:50.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049c00 con 0x7f21cc005e30 2021-03-09T10:00:50.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4003cb0 2021-03-09T10:00:50.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:50.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fce0 con 0x7f21cc02f1a0 2021-03-09T10:00:50.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:50.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:50.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:50.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:50.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:50.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:50.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:50.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:50.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046540 con 0x7f21cc02f1a0 2021-03-09T10:00:50.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0533a0 con 0x7f21cc02f1a0 2021-03-09T10:00:50.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8451 cs=0 l=1 rev1=1 rx=0x7f21cc01baf0 tx=0x7f21e404f7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:50.289 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:50.279+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28623 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:50.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.279+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4047210 con 0x7f21cc02f1a0 2021-03-09T10:00:50.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.279+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4047370 con 0x7f21cc02f1a0 2021-03-09T10:00:50.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4047210 con 0x7f21cc02f1a0 2021-03-09T10:00:50.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.287+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:50.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.287+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:50.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:50.287+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:00:53.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:53.699 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:00:55.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8451 cs=0 l=1 rev1=1 rx=0x7f21cc01baf0 tx=0x7f21e404f7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8451 cs=0 l=1 rev1=1 rx=0x7f21cc01baf0 tx=0x7f21e404f7b0).stop 2021-03-09T10:00:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049c00 con 0x7f21cc005e30 2021-03-09T10:00:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046bc0 con 0x7f21c4003cb0 2021-03-09T10:00:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4008860 2021-03-09T10:00:55.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:00:55.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:55.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:00:55.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:55.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:55.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052eb0 con 0x7f21c4008860 2021-03-09T10:00:55.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21c4008860 2021-03-09T10:00:55.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 secure :-1 s=READY pgs=8464 cs=0 l=1 rev1=1 rx=0x7f21dc009cf0 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:55.293 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:00:55.283+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28661 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:00:55.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07ff90 con 0x7f21c4008860 2021-03-09T10:00:55.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.291+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0332f0 con 0x7f21c4008860 2021-03-09T10:00:55.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.291+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21c4008860 2021-03-09T10:00:55.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.291+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:00:55.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.295+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:00:55.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:00:55.295+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:00.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:00.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:00.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 secure :-1 s=READY pgs=8464 cs=0 l=1 rev1=1 rx=0x7f21dc009cf0 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:00.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 secure :-1 s=READY pgs=8464 cs=0 l=1 rev1=1 rx=0x7f21dc009cf0 tx=0x7f21dc009880).stop 2021-03-09T10:01:00.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0466d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:00.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 unknown :-1 s=CLOSED pgs=8464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:00.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:00.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:00.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:00.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046bc0 con 0x7f21cc005e30 2021-03-09T10:01:00.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d970 con 0x7f21c4003cb0 2021-03-09T10:01:00.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21cc02f1a0 2021-03-09T10:01:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.287+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:00.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:00.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:00.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:00.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0983f0 con 0x7f21cc02f1a0 2021-03-09T10:01:00.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052eb0 con 0x7f21cc02f1a0 2021-03-09T10:01:00.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8473 cs=0 l=1 rev1=1 rx=0x7f21dc0697c0 tx=0x7f21dc0333d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:00.300 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:00.291+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28697 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21cc02f1a0 2021-03-09T10:01:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032a60 con 0x7f21cc02f1a0 2021-03-09T10:01:00.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21cc02f1a0 2021-03-09T10:01:00.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.299+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:00.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:00.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:00.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:05.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:05.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:05.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.291+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8473 cs=0 l=1 rev1=1 rx=0x7f21dc0697c0 tx=0x7f21dc0333d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:05.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8473 cs=0 l=1 rev1=1 rx=0x7f21dc0697c0 tx=0x7f21dc0333d0).stop 2021-03-09T10:01:05.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:05.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:05.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:05.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:05.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:05.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046bc0 con 0x7f21cc005e30 2021-03-09T10:01:05.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057c10 con 0x7f21c4003cb0 2021-03-09T10:01:05.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4008860 2021-03-09T10:01:05.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:05.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:05.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:05.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:05.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:05.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:05.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:05.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:05.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:05.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052eb0 con 0x7f21c4008860 2021-03-09T10:01:05.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0983f0 con 0x7f21c4008860 2021-03-09T10:01:05.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 secure :-1 s=READY pgs=8483 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:05.304 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:05.299+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:05.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21c4008860 2021-03-09T10:01:05.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c310 con 0x7f21c4008860 2021-03-09T10:01:05.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21c4008860 2021-03-09T10:01:05.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.303+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:05.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:05.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:05.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:10.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:10.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:10.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 secure :-1 s=READY pgs=8483 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:10.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 secure :-1 s=READY pgs=8483 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010d90).stop 2021-03-09T10:01:10.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0466d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:10.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0466d0 unknown :-1 s=CLOSED pgs=8483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:10.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:10.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:10.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:10.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046bc0 con 0x7f21cc005e30 2021-03-09T10:01:10.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4003cb0 2021-03-09T10:01:10.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054c30 con 0x7f21cc02f1a0 2021-03-09T10:01:10.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:10.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:10.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:10.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:10.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:10.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:10.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:10.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:10.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:10.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc054da0 con 0x7f21cc02f1a0 2021-03-09T10:01:10.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.299+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046540 con 0x7f21cc02f1a0 2021-03-09T10:01:10.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8494 cs=0 l=1 rev1=1 rx=0x7f21e40072f0 tx=0x7f21e402bf80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:10.308 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:10.303+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28770 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:10.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T10:01:10.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d50 con 0x7f21cc02f1a0 2021-03-09T10:01:10.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21cc02f1a0 2021-03-09T10:01:10.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.307+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:10.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.307+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:10.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:10.307+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:15.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:15.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:15.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8494 cs=0 l=1 rev1=1 rx=0x7f21e40072f0 tx=0x7f21e402bf80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:15.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8494 cs=0 l=1 rev1=1 rx=0x7f21e40072f0 tx=0x7f21e402bf80).stop 2021-03-09T10:01:15.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:15.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:15.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc053040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:15.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:15.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:15.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21cc005e30 2021-03-09T10:01:15.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc054c00 con 0x7f21c4003cb0 2021-03-09T10:01:15.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0464f0 con 0x7f21c4008860 2021-03-09T10:01:15.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:15.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:15.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:15.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:15.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.303+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc053040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:15.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:15.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:15.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:15.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:15.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046630 con 0x7f21c4008860 2021-03-09T10:01:15.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052eb0 con 0x7f21c4008860 2021-03-09T10:01:15.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc053040 secure :-1 s=READY pgs=8504 cs=0 l=1 rev1=1 rx=0x7f21dc029f10 tx=0x7f21dc003fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:15.313 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28808 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:15.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05b1d0 con 0x7f21c4008860 2021-03-09T10:01:15.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc078e70 con 0x7f21c4008860 2021-03-09T10:01:15.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T10:01:15.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.315+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:15.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.315+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:15.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:15.315+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:20.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc053040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:20.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc053040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:20.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc053040 secure :-1 s=READY pgs=8504 cs=0 l=1 rev1=1 rx=0x7f21dc029f10 tx=0x7f21dc003fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:20.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc053040 secure :-1 s=READY pgs=8504 cs=0 l=1 rev1=1 rx=0x7f21dc029f10 tx=0x7f21dc003fd0).stop 2021-03-09T10:01:20.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc053040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:20.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc053040 unknown :-1 s=CLOSED pgs=8504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:20.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:20.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:20.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:20.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21cc005e30 2021-03-09T10:01:20.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21c4003cb0 2021-03-09T10:01:20.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21cc02f1a0 2021-03-09T10:01:20.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:20.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:20.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:20.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:20.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:20.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:20.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:20.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:20.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:20.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0464a0 con 0x7f21cc02f1a0 2021-03-09T10:01:20.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc054da0 con 0x7f21cc02f1a0 2021-03-09T10:01:20.318 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:20.311+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28844 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:20.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8515 cs=0 l=1 rev1=1 rx=0x7f21cc022220 tx=0x7f21e4059000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:20.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T10:01:20.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21cc02f1a0 2021-03-09T10:01:20.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc02f1a0 2021-03-09T10:01:20.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.319+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:20.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.319+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:20.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:20.319+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:23.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:23.755 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:25.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:01:25.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:25.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8515 cs=0 l=1 rev1=1 rx=0x7f21cc022220 tx=0x7f21e4059000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:25.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8515 cs=0 l=1 rev1=1 rx=0x7f21cc022220 tx=0x7f21e4059000).stop 2021-03-09T10:01:25.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:25.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:25.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:25.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:25.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:25.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21cc005e30 2021-03-09T10:01:25.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007cc0 con 0x7f21c4003cb0 2021-03-09T10:01:25.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21c4008860 2021-03-09T10:01:25.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:25.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:25.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:25.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:25.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:25.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:25.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:25.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:25.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:25.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052ff0 con 0x7f21c4008860 2021-03-09T10:01:25.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.315+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0464a0 con 0x7f21c4008860 2021-03-09T10:01:25.323 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:25.319+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28880 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:25.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0467c0 secure :-1 s=READY pgs=8527 cs=0 l=1 rev1=1 rx=0x7f21cc0952e0 tx=0x7f21e407d1a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:25.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027eb0 con 0x7f21c4008860 2021-03-09T10:01:25.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027eb0 con 0x7f21c4008860 2021-03-09T10:01:25.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c4008860 2021-03-09T10:01:25.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.327+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:25.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.327+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:25.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:25.327+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:30.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0467c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:30.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0467c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:30.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0467c0 secure :-1 s=READY pgs=8527 cs=0 l=1 rev1=1 rx=0x7f21cc0952e0 tx=0x7f21e407d1a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:30.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0467c0 secure :-1 s=READY pgs=8527 cs=0 l=1 rev1=1 rx=0x7f21cc0952e0 tx=0x7f21e407d1a0).stop 2021-03-09T10:01:30.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc0467c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:30.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc0467c0 unknown :-1 s=CLOSED pgs=8527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:30.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:30.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:30.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:30.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052eb0 con 0x7f21cc005e30 2021-03-09T10:01:30.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0471d0 con 0x7f21c4003cb0 2021-03-09T10:01:30.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f6d0 con 0x7f21cc02f1a0 2021-03-09T10:01:30.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:30.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:30.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:30.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:30.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.319+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:30.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:30.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:30.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:30.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:30.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0464a0 con 0x7f21cc02f1a0 2021-03-09T10:01:30.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046630 con 0x7f21cc02f1a0 2021-03-09T10:01:30.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8540 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:30.328 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:30.323+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28918 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:30.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc02f1a0 2021-03-09T10:01:30.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032150 con 0x7f21cc02f1a0 2021-03-09T10:01:30.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc02f1a0 2021-03-09T10:01:30.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.331+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:30.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:30.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:30.331+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:35.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:35.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:35.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8540 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:35.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8540 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc077e50).stop 2021-03-09T10:01:35.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:35.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:35.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:35.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:35.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:35.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046b20 con 0x7f21cc005e30 2021-03-09T10:01:35.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047160 con 0x7f21c4003cb0 2021-03-09T10:01:35.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050d00 con 0x7f21c4008860 2021-03-09T10:01:35.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:35.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:35.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:35.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:35.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:35.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:35.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:35.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:35.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:35.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc050e70 con 0x7f21c4008860 2021-03-09T10:01:35.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0464a0 con 0x7f21c4008860 2021-03-09T10:01:35.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.327+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046630 secure :-1 s=READY pgs=8549 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0abee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:35.335 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:35.331+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28954 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:35.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21c4008860 2021-03-09T10:01:35.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c310 con 0x7f21c4008860 2021-03-09T10:01:35.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T10:01:35.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.339+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:35.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.339+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:35.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:35.339+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:40.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:40.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:40.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046630 secure :-1 s=READY pgs=8549 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0abee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:40.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046630 secure :-1 s=READY pgs=8549 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc0abee0).stop 2021-03-09T10:01:40.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:40.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046630 unknown :-1 s=CLOSED pgs=8549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:40.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:40.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:40.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:40.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013490 con 0x7f21cc005e30 2021-03-09T10:01:40.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046b20 con 0x7f21c4003cb0 2021-03-09T10:01:40.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0462b0 con 0x7f21cc02f1a0 2021-03-09T10:01:40.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:40.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:40.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:40.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:40.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:40.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:40.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:40.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:40.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:40.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc02f1a0 2021-03-09T10:01:40.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc050e70 con 0x7f21cc02f1a0 2021-03-09T10:01:40.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8558 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc029610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:40.341 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:40.334+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28992 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:40.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc02f1a0 2021-03-09T10:01:40.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc02f1a0 2021-03-09T10:01:40.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc02f1a0 2021-03-09T10:01:40.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.342+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:40.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:40.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:40.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:45.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:45.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:45.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8558 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc029610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:45.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 secure :-1 s=READY pgs=8558 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc029610).stop 2021-03-09T10:01:45.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0158d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:45.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0158d0 unknown :-1 s=CLOSED pgs=8558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:45.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:45.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:45.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:45.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013630 con 0x7f21cc005e30 2021-03-09T10:01:45.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046260 con 0x7f21c4003cb0 2021-03-09T10:01:45.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046a70 con 0x7f21c4008860 2021-03-09T10:01:45.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:45.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:45.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:45.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:45.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:45.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:45.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:45.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:45.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:45.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053180 con 0x7f21c4008860 2021-03-09T10:01:45.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052ff0 con 0x7f21c4008860 2021-03-09T10:01:45.345 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29029 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:45.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 secure :-1 s=READY pgs=8569 cs=0 l=1 rev1=1 rx=0x7f21cc01d3a0 tx=0x7f21e4016f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:45.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031230 con 0x7f21c4008860 2021-03-09T10:01:45.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T10:01:45.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T10:01:45.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.346+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:45.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:45.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:45.346+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:50.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:50.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:50.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 secure :-1 s=READY pgs=8569 cs=0 l=1 rev1=1 rx=0x7f21cc01d3a0 tx=0x7f21e4016f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:50.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 secure :-1 s=READY pgs=8569 cs=0 l=1 rev1=1 rx=0x7f21cc01d3a0 tx=0x7f21e4016f20).stop 2021-03-09T10:01:50.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:50.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 unknown :-1 s=CLOSED pgs=8569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:50.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:50.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:50.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:50.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013630 con 0x7f21cc005e30 2021-03-09T10:01:50.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dad0 con 0x7f21c4003cb0 2021-03-09T10:01:50.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040020 con 0x7f21cc02f1a0 2021-03-09T10:01:50.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:50.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:50.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:50.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:50.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:50.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:50.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:50.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:50.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:50.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052ff0 con 0x7f21cc02f1a0 2021-03-09T10:01:50.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053180 con 0x7f21cc02f1a0 2021-03-09T10:01:50.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.342+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8579 cs=0 l=1 rev1=1 rx=0x7f21e4027ff0 tx=0x7f21e4028f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:50.364 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:50.346+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29067 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:50.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a1d30 con 0x7f21cc02f1a0 2021-03-09T10:01:50.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4031020 con 0x7f21cc02f1a0 2021-03-09T10:01:50.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a1d30 con 0x7f21cc02f1a0 2021-03-09T10:01:50.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.350+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:50.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:50.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:50.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:01:53.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:53.802 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:01:55.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:55.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:55.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8579 cs=0 l=1 rev1=1 rx=0x7f21e4027ff0 tx=0x7f21e4028f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:55.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8579 cs=0 l=1 rev1=1 rx=0x7f21e4027ff0 tx=0x7f21e4028f40).stop 2021-03-09T10:01:55.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:55.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=CLOSED pgs=8579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:55.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:55.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:55.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:55.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dad0 con 0x7f21cc005e30 2021-03-09T10:01:55.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046a70 con 0x7f21c4003cb0 2021-03-09T10:01:55.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046bb0 con 0x7f21c4008860 2021-03-09T10:01:55.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:01:55.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:55.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:01:55.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:55.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:55.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:55.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:55.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:55.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:55.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03fe80 con 0x7f21c4008860 2021-03-09T10:01:55.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052ff0 con 0x7f21c4008860 2021-03-09T10:01:55.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 secure :-1 s=READY pgs=8590 cs=0 l=1 rev1=1 rx=0x7f21cc035f80 tx=0x7f21e40a6fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:55.364 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:01:55.350+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29104 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:01:55.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a6d80 con 0x7f21c4008860 2021-03-09T10:01:55.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401f250 con 0x7f21c4008860 2021-03-09T10:01:55.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a1d30 con 0x7f21c4008860 2021-03-09T10:01:55.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.358+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:01:55.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:01:55.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:01:55.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:00.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:00.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:00.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 secure :-1 s=READY pgs=8590 cs=0 l=1 rev1=1 rx=0x7f21cc035f80 tx=0x7f21e40a6fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:00.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 secure :-1 s=READY pgs=8590 cs=0 l=1 rev1=1 rx=0x7f21cc035f80 tx=0x7f21e40a6fd0).stop 2021-03-09T10:02:00.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc046580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:00.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc046580 unknown :-1 s=CLOSED pgs=8590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:00.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:00.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:00.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:00.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015d20 con 0x7f21cc005e30 2021-03-09T10:02:00.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031b00 con 0x7f21c4003cb0 2021-03-09T10:02:00.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc031c40 con 0x7f21cc02f1a0 2021-03-09T10:02:00.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:00.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:00.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:00.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:00.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:00.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:00.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:00.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:00.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:00.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc031db0 con 0x7f21cc02f1a0 2021-03-09T10:02:00.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03fe80 con 0x7f21cc02f1a0 2021-03-09T10:02:00.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8601 cs=0 l=1 rev1=1 rx=0x7f21e40b02e0 tx=0x7f21e4033cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:00.362 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:00.354+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29142 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:00.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401f020 con 0x7f21cc02f1a0 2021-03-09T10:02:00.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401f020 con 0x7f21cc02f1a0 2021-03-09T10:02:00.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401f000 con 0x7f21cc02f1a0 2021-03-09T10:02:00.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.366+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:00.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.366+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:00.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:00.366+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:05.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:05.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:05.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8601 cs=0 l=1 rev1=1 rx=0x7f21e40b02e0 tx=0x7f21e4033cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:05.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8601 cs=0 l=1 rev1=1 rx=0x7f21e40b02e0 tx=0x7f21e4033cf0).stop 2021-03-09T10:02:05.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:05.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=CLOSED pgs=8601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:05.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:05.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:05.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:05.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015d20 con 0x7f21cc005e30 2021-03-09T10:02:05.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040020 con 0x7f21c4003cb0 2021-03-09T10:02:05.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046580 con 0x7f21c4008860 2021-03-09T10:02:05.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:05.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:05.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:05.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:05.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:05.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:05.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:05.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:05.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:05.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0466c0 con 0x7f21c4008860 2021-03-09T10:02:05.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0463f0 con 0x7f21c4008860 2021-03-09T10:02:05.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.358+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8611 cs=0 l=1 rev1=1 rx=0x7f21cc027110 tx=0x7f21dc029300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:05.368 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:05.362+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:05.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0294a0 con 0x7f21c4008860 2021-03-09T10:02:05.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21c4008860 2021-03-09T10:02:05.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T10:02:05.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.366+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:05.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.366+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:05.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:05.366+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:10.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:10.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:10.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8611 cs=0 l=1 rev1=1 rx=0x7f21cc027110 tx=0x7f21dc029300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:10.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8611 cs=0 l=1 rev1=1 rx=0x7f21cc027110 tx=0x7f21dc029300).stop 2021-03-09T10:02:10.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:10.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:10.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:10.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:10.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:10.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036300 con 0x7f21cc005e30 2021-03-09T10:02:10.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046550 con 0x7f21c4003cb0 2021-03-09T10:02:10.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:10.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:10.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040020 con 0x7f21cc02f1a0 2021-03-09T10:02:10.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:10.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:10.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:10.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:10.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:10.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:10.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:10.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015d20 con 0x7f21cc02f1a0 2021-03-09T10:02:10.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0466c0 con 0x7f21cc02f1a0 2021-03-09T10:02:10.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8623 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc021180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:10.372 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:10.366+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29217 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:10.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b6330 con 0x7f21cc02f1a0 2021-03-09T10:02:10.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc02f1a0 2021-03-09T10:02:10.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b6120 con 0x7f21cc02f1a0 2021-03-09T10:02:10.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.370+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:10.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:10.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:10.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:15.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:15.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:15.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8623 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc021180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:15.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 secure :-1 s=READY pgs=8623 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc021180).stop 2021-03-09T10:02:15.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc015830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:15.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc015830 unknown :-1 s=CLOSED pgs=8623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:15.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:15.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:15.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:15.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046a40 con 0x7f21cc005e30 2021-03-09T10:02:15.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046550 con 0x7f21c4003cb0 2021-03-09T10:02:15.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015eb0 con 0x7f21c4008860 2021-03-09T10:02:15.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:15.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:15.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:15.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:15.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.366+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:15.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:15.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:15.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:15.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:15.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022d60 con 0x7f21c4008860 2021-03-09T10:02:15.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc031db0 con 0x7f21c4008860 2021-03-09T10:02:15.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8635 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4028ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:15.377 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:15.370+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29253 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:15.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21c4008860 2021-03-09T10:02:15.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21c4008860 2021-03-09T10:02:15.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T10:02:15.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.378+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:15.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.378+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:15.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:15.378+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:20.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:20.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:20.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8635 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4028ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:20.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8635 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4028ca0).stop 2021-03-09T10:02:20.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:20.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:20.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:20.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:20.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0157c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:20.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a0b0 con 0x7f21cc005e30 2021-03-09T10:02:20.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0564f0 con 0x7f21c4003cb0 2021-03-09T10:02:20.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b8d0 con 0x7f21cc02f1a0 2021-03-09T10:02:20.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:20.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:20.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:20.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:20.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:20.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:20.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:20.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:20.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:20.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015eb0 con 0x7f21cc02f1a0 2021-03-09T10:02:20.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015d20 con 0x7f21cc02f1a0 2021-03-09T10:02:20.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0157c0 secure :-1 s=READY pgs=8643 cs=0 l=1 rev1=1 rx=0x7f21cc011f70 tx=0x7f21dc003fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:20.383 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:20.378+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29292 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:20.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc02f1a0 2021-03-09T10:02:20.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc02f1a0 2021-03-09T10:02:20.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc02f1a0 2021-03-09T10:02:20.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.382+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:20.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.382+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:20.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:20.382+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:23.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:23.845 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:25.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0157c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:25.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0157c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:25.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0157c0 secure :-1 s=READY pgs=8643 cs=0 l=1 rev1=1 rx=0x7f21cc011f70 tx=0x7f21dc003fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:25.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0157c0 secure :-1 s=READY pgs=8643 cs=0 l=1 rev1=1 rx=0x7f21cc011f70 tx=0x7f21dc003fb0).stop 2021-03-09T10:02:25.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0157c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:25.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0157c0 unknown :-1 s=CLOSED pgs=8643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:25.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:25.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:25.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:25.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057ea0 con 0x7f21cc005e30 2021-03-09T10:02:25.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057fe0 con 0x7f21c4003cb0 2021-03-09T10:02:25.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0466c0 con 0x7f21c4008860 2021-03-09T10:02:25.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:25.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:25.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:25.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:25.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:25.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:25.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:25.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:25.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:25.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046a00 con 0x7f21c4008860 2021-03-09T10:02:25.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022d60 con 0x7f21c4008860 2021-03-09T10:02:25.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8654 cs=0 l=1 rev1=1 rx=0x7f21cc01b610 tx=0x7f21e4059070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:25.387 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:25.382+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29328 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:25.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007e00 con 0x7f21c4008860 2021-03-09T10:02:25.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4031290 con 0x7f21c4008860 2021-03-09T10:02:25.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4008860 2021-03-09T10:02:25.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.390+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:25.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:25.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:25.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:30.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:30.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:30.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8654 cs=0 l=1 rev1=1 rx=0x7f21cc01b610 tx=0x7f21e4059070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:30.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8654 cs=0 l=1 rev1=1 rx=0x7f21cc01b610 tx=0x7f21e4059070).stop 2021-03-09T10:02:30.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:30.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:30.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:30.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:30.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:30.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058010 con 0x7f21cc005e30 2021-03-09T10:02:30.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0466c0 con 0x7f21c4003cb0 2021-03-09T10:02:30.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022bc0 con 0x7f21cc02f1a0 2021-03-09T10:02:30.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:30.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:30.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:30.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:30.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:30.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:30.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:30.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:30.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:30.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022d00 con 0x7f21cc02f1a0 2021-03-09T10:02:30.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc015eb0 con 0x7f21cc02f1a0 2021-03-09T10:02:30.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8666 cs=0 l=1 rev1=1 rx=0x7f21cc017ed0 tx=0x7f21dc029240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:30.395 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:30.386+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29367 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:30.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc02f1a0 2021-03-09T10:02:30.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc02f1a0 2021-03-09T10:02:30.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.394+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc02f1a0 2021-03-09T10:02:30.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.394+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:30.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.394+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:30.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:30.394+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:35.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:35.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:35.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8666 cs=0 l=1 rev1=1 rx=0x7f21cc017ed0 tx=0x7f21dc029240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:35.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8666 cs=0 l=1 rev1=1 rx=0x7f21cc017ed0 tx=0x7f21dc029240).stop 2021-03-09T10:02:35.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:35.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=CLOSED pgs=8666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:35.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:35.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:35.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:35.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022b70 con 0x7f21cc005e30 2021-03-09T10:02:35.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013c60 con 0x7f21c4003cb0 2021-03-09T10:02:35.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013da0 con 0x7f21c4008860 2021-03-09T10:02:35.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:35.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:35.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:35.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:35.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:35.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:35.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:35.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:35.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:35.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013ee0 con 0x7f21c4008860 2021-03-09T10:02:35.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046a00 con 0x7f21c4008860 2021-03-09T10:02:35.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8676 cs=0 l=1 rev1=1 rx=0x7f21e40b1290 tx=0x7f21e4003a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:35.400 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:35.393+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29403 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:35.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016eb0 con 0x7f21c4008860 2021-03-09T10:02:35.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21c4008860 2021-03-09T10:02:35.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21c4008860 2021-03-09T10:02:35.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.397+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:35.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:35.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:35.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:40.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:40.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:40.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8676 cs=0 l=1 rev1=1 rx=0x7f21e40b1290 tx=0x7f21e4003a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:40.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8676 cs=0 l=1 rev1=1 rx=0x7f21e40b1290 tx=0x7f21e4003a10).stop 2021-03-09T10:02:40.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:40.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:40.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:40.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:40.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:40.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013c60 con 0x7f21cc005e30 2021-03-09T10:02:40.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013da0 con 0x7f21c4003cb0 2021-03-09T10:02:40.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043300 con 0x7f21cc02f1a0 2021-03-09T10:02:40.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:40.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:40.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:40.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:40.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:40.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:40.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:40.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:40.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:40.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043440 con 0x7f21cc02f1a0 2021-03-09T10:02:40.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022d00 con 0x7f21cc02f1a0 2021-03-09T10:02:40.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8687 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:40.404 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:40.397+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29442 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:40.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc02f1a0 2021-03-09T10:02:40.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc02f1a0 2021-03-09T10:02:40.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T10:02:40.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.405+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:40.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.409+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:40.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:40.409+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:45.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:45.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:45.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8687 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:45.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8687 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077e50).stop 2021-03-09T10:02:45.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:45.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=CLOSED pgs=8687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:45.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:45.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:45.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:45.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057750 con 0x7f21cc005e30 2021-03-09T10:02:45.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048b90 con 0x7f21c4003cb0 2021-03-09T10:02:45.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048cd0 con 0x7f21c4008860 2021-03-09T10:02:45.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:45.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:45.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:45.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:45.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:45.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:45.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:45.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:45.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:45.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095050 con 0x7f21c4008860 2021-03-09T10:02:45.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.401+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013ee0 con 0x7f21c4008860 2021-03-09T10:02:45.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8697 cs=0 l=1 rev1=1 rx=0x7f21e40554c0 tx=0x7f21e4055560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:45.411 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:45.405+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29478 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:45.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050090 con 0x7f21c4008860 2021-03-09T10:02:45.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055330 con 0x7f21c4008860 2021-03-09T10:02:45.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c4008860 2021-03-09T10:02:45.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.413+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:45.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:45.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:45.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:50.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:02:50.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:50.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8697 cs=0 l=1 rev1=1 rx=0x7f21e40554c0 tx=0x7f21e4055560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:50.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8697 cs=0 l=1 rev1=1 rx=0x7f21e40554c0 tx=0x7f21e4055560).stop 2021-03-09T10:02:50.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:50.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:50.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:50.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:50.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:50.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009a30 con 0x7f21cc005e30 2021-03-09T10:02:50.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055010 con 0x7f21c4003cb0 2021-03-09T10:02:50.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0951e0 con 0x7f21cc02f1a0 2021-03-09T10:02:50.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:50.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:50.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:50.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:50.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:50.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:50.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:50.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:50.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:50.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013c60 con 0x7f21cc02f1a0 2021-03-09T10:02:50.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043440 con 0x7f21cc02f1a0 2021-03-09T10:02:50.417 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29517 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:50.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8711 cs=0 l=1 rev1=1 rx=0x7f21dc0196c0 tx=0x7f21dc07cc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:50.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc02f1a0 2021-03-09T10:02:50.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc019520 con 0x7f21cc02f1a0 2021-03-09T10:02:50.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T10:02:50.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.417+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:50.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.417+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:50.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:50.417+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:02:53.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:53.892 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:02:55.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:55.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:55.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8711 cs=0 l=1 rev1=1 rx=0x7f21dc0196c0 tx=0x7f21dc07cc20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:55.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 secure :-1 s=READY pgs=8711 cs=0 l=1 rev1=1 rx=0x7f21dc0196c0 tx=0x7f21dc07cc20).stop 2021-03-09T10:02:55.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc013770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:55.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc013770 unknown :-1 s=CLOSED pgs=8711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:55.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:55.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:55.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:55.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0951e0 con 0x7f21cc005e30 2021-03-09T10:02:55.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009a30 con 0x7f21c4003cb0 2021-03-09T10:02:55.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0125b0 con 0x7f21c4008860 2021-03-09T10:02:55.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:02:55.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:55.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:02:55.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:55.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:55.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:55.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:55.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:55.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:55.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013c60 con 0x7f21c4008860 2021-03-09T10:02:55.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043440 con 0x7f21c4008860 2021-03-09T10:02:55.422 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:02:55.413+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:02:55.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8720 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc0571d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:55.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T10:02:55.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4008860 2021-03-09T10:02:55.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21c4008860 2021-03-09T10:02:55.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.421+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:02:55.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:02:55.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:02:55.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:00.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8720 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc0571d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:00.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8720 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc0571d0).stop 2021-03-09T10:03:00.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:00.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d1b0 con 0x7f21cc005e30 2021-03-09T10:03:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d2f0 con 0x7f21c4003cb0 2021-03-09T10:03:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d430 con 0x7f21cc02f1a0 2021-03-09T10:03:00.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:00.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:00.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:00.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:00.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:00.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:00.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:00.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d570 con 0x7f21cc02f1a0 2021-03-09T10:03:00.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043440 con 0x7f21cc02f1a0 2021-03-09T10:03:00.427 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29592 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:00.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8732 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc019760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:00.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc057e70 con 0x7f21cc02f1a0 2021-03-09T10:03:00.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc02f1a0 2021-03-09T10:03:00.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bb230 con 0x7f21cc02f1a0 2021-03-09T10:03:00.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.425+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:00.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.429+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:00.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:00.429+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:05.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:05.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:05.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8732 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc019760).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:05.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8732 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc019760).stop 2021-03-09T10:03:05.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:05.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:05.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:05.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035ac0 con 0x7f21cc005e30 2021-03-09T10:03:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035c00 con 0x7f21c4003cb0 2021-03-09T10:03:05.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d1b0 con 0x7f21c4008860 2021-03-09T10:03:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:05.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:05.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.421+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:05.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:05.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:05.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:05.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:05.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d2f0 con 0x7f21c4008860 2021-03-09T10:03:05.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095050 con 0x7f21c4008860 2021-03-09T10:03:05.435 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29628 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:05.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8742 cs=0 l=1 rev1=1 rx=0x7f21e40a7060 tx=0x7f21e4003a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:05.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050070 con 0x7f21c4008860 2021-03-09T10:03:05.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21c4008860 2021-03-09T10:03:05.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4008860 2021-03-09T10:03:05.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.433+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:05.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.433+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:05.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:05.433+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:10.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:03:10.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:10.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8742 cs=0 l=1 rev1=1 rx=0x7f21e40a7060 tx=0x7f21e4003a80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:10.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 secure :-1 s=READY pgs=8742 cs=0 l=1 rev1=1 rx=0x7f21e40a7060 tx=0x7f21e4003a80).stop 2021-03-09T10:03:10.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc052ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:10.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc052ff0 unknown :-1 s=CLOSED pgs=8742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:10.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:10.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:10.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:10.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d1b0 con 0x7f21cc005e30 2021-03-09T10:03:10.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035c30 con 0x7f21c4003cb0 2021-03-09T10:03:10.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012180 con 0x7f21cc02f1a0 2021-03-09T10:03:10.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:10.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:10.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:10.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:10.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:10.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:10.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:10.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:10.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:10.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095000 con 0x7f21cc02f1a0 2021-03-09T10:03:10.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d2f0 con 0x7f21cc02f1a0 2021-03-09T10:03:10.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e40a1f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:10.438 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:10.433+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29667 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:10.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050070 con 0x7f21cc02f1a0 2021-03-09T10:03:10.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407ed50 con 0x7f21cc02f1a0 2021-03-09T10:03:10.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc02f1a0 2021-03-09T10:03:10.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.437+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:10.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.437+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:10.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:10.437+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:15.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:15.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:15.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e40a1f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:15.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f21e4016ff0 tx=0x7f21e40a1f60).stop 2021-03-09T10:03:15.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:15.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:15.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:15.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:15.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:15.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:15.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:15.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d350 con 0x7f21cc005e30 2021-03-09T10:03:15.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052ff0 con 0x7f21c4003cb0 2021-03-09T10:03:15.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053130 con 0x7f21c4008860 2021-03-09T10:03:15.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:15.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:15.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.433+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:15.442 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29703 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:15.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:15.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:15.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:15.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:15.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053270 con 0x7f21c4008860 2021-03-09T10:03:15.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d570 con 0x7f21c4008860 2021-03-09T10:03:15.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047550 secure :-1 s=READY pgs=8767 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc062f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:15.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00d590 con 0x7f21c4008860 2021-03-09T10:03:15.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062310 con 0x7f21c4008860 2021-03-09T10:03:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4008860 2021-03-09T10:03:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.445+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:15.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.445+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:15.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:15.445+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:20.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:20.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:20.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047550 secure :-1 s=READY pgs=8767 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc062f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:20.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047550 secure :-1 s=READY pgs=8767 cs=0 l=1 rev1=1 rx=0x7f21dc081380 tx=0x7f21dc062f30).stop 2021-03-09T10:03:20.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:20.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047550 unknown :-1 s=CLOSED pgs=8767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:20.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:20.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:20.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:20.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cac0 con 0x7f21cc005e30 2021-03-09T10:03:20.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc00 con 0x7f21c4003cb0 2021-03-09T10:03:20.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cd40 con 0x7f21cc02f1a0 2021-03-09T10:03:20.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:20.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:20.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:20.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:20.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:20.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:20.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:20.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:20.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:20.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013ea0 con 0x7f21cc02f1a0 2021-03-09T10:03:20.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053270 con 0x7f21cc02f1a0 2021-03-09T10:03:20.449 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:20.441+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29742 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:20.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8776 cs=0 l=1 rev1=1 rx=0x7f21dc072360 tx=0x7f21dc010e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:20.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc02f1a0 2021-03-09T10:03:20.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T10:03:20.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc02f1a0 2021-03-09T10:03:20.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.449+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:20.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.449+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:20.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:20.449+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:23.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:23.939 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:25.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:25.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:25.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8776 cs=0 l=1 rev1=1 rx=0x7f21dc072360 tx=0x7f21dc010e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:25.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8776 cs=0 l=1 rev1=1 rx=0x7f21dc072360 tx=0x7f21dc010e00).stop 2021-03-09T10:03:25.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:25.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:25.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:25.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:25.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:25.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fc0 con 0x7f21cc005e30 2021-03-09T10:03:25.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f780 con 0x7f21c4003cb0 2021-03-09T10:03:25.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ce80 con 0x7f21c4008860 2021-03-09T10:03:25.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:25.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:25.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:25.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:25.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:25.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:25.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:25.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:25.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:25.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047850 con 0x7f21c4008860 2021-03-09T10:03:25.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013ea0 con 0x7f21c4008860 2021-03-09T10:03:25.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8782 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:25.454 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:25.445+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29778 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:25.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b10a0 con 0x7f21c4008860 2021-03-09T10:03:25.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21c4008860 2021-03-09T10:03:25.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b6d40 con 0x7f21c4008860 2021-03-09T10:03:25.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.453+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:25.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.453+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:25.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:25.453+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:30.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:30.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:30.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8782 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:30.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8782 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc009810).stop 2021-03-09T10:03:30.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:30.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=CLOSED pgs=8782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:30.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:30.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:30.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:30.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dfd0 con 0x7f21cc005e30 2021-03-09T10:03:30.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ce80 con 0x7f21c4003cb0 2021-03-09T10:03:30.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03f780 con 0x7f21cc02f1a0 2021-03-09T10:03:30.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:30.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:30.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:30.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:30.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:30.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:30.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:30.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:30.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:30.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03f8c0 con 0x7f21cc02f1a0 2021-03-09T10:03:30.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc047850 con 0x7f21cc02f1a0 2021-03-09T10:03:30.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.448+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8792 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc08a120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:30.461 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:30.452+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29817 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:30.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02ae60 con 0x7f21cc02f1a0 2021-03-09T10:03:30.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3950 con 0x7f21cc02f1a0 2021-03-09T10:03:30.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc02f1a0 2021-03-09T10:03:30.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.456+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:30.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.456+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:30.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:30.456+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:35.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:35.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:35.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8792 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc08a120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:35.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8792 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc08a120).stop 2021-03-09T10:03:35.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:35.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:35.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:35.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:35.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:35.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d10 con 0x7f21cc005e30 2021-03-09T10:03:35.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047850 con 0x7f21c4003cb0 2021-03-09T10:03:35.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04de30 con 0x7f21c4008860 2021-03-09T10:03:35.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:35.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:35.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:35.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:35.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:35.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:35.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:35.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:35.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:35.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04df70 con 0x7f21c4008860 2021-03-09T10:03:35.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03f8c0 con 0x7f21c4008860 2021-03-09T10:03:35.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8803 cs=0 l=1 rev1=1 rx=0x7f21dc097c10 tx=0x7f21dc0724f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:35.465 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:35.456+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:35.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.460+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4008860 2021-03-09T10:03:35.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.460+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083380 con 0x7f21c4008860 2021-03-09T10:03:35.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T10:03:35.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.464+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:35.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.464+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:35.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:35.464+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:40.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:40.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:40.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8803 cs=0 l=1 rev1=1 rx=0x7f21dc097c10 tx=0x7f21dc0724f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:40.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8803 cs=0 l=1 rev1=1 rx=0x7f21dc097c10 tx=0x7f21dc0724f0).stop 2021-03-09T10:03:40.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:40.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=CLOSED pgs=8803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:40.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:40.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:40.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:40.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d10 con 0x7f21cc005e30 2021-03-09T10:03:40.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047850 con 0x7f21c4003cb0 2021-03-09T10:03:40.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cd00 con 0x7f21cc02f1a0 2021-03-09T10:03:40.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:40.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:40.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:40.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.456+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:40.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:40.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:40.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:40.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:40.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:40.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013ea0 con 0x7f21cc02f1a0 2021-03-09T10:03:40.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04df70 con 0x7f21cc02f1a0 2021-03-09T10:03:40.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.460+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8814 cs=0 l=1 rev1=1 rx=0x7f21dc0725d0 tx=0x7f21dc083150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:40.471 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:40.464+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29892 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:40.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3950 con 0x7f21cc02f1a0 2021-03-09T10:03:40.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc091880 con 0x7f21cc02f1a0 2021-03-09T10:03:40.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T10:03:40.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.468+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:40.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.468+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:40.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:40.468+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:45.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:45.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:45.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8814 cs=0 l=1 rev1=1 rx=0x7f21dc0725d0 tx=0x7f21dc083150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:45.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8814 cs=0 l=1 rev1=1 rx=0x7f21dc0725d0 tx=0x7f21dc083150).stop 2021-03-09T10:03:45.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:45.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:45.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:45.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:45.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:45.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016040 con 0x7f21cc005e30 2021-03-09T10:03:45.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0860a0 con 0x7f21c4003cb0 2021-03-09T10:03:45.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047850 con 0x7f21c4008860 2021-03-09T10:03:45.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:45.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:45.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:45.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:45.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.464+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:45.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:45.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:45.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:45.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:45.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00fb60 con 0x7f21c4008860 2021-03-09T10:03:45.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095000 con 0x7f21c4008860 2021-03-09T10:03:45.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8827 cs=0 l=1 rev1=1 rx=0x7f21e40a6040 tx=0x7f21e4003b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:45.478 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:45.468+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29928 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:45.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050050 con 0x7f21c4008860 2021-03-09T10:03:45.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21c4008860 2021-03-09T10:03:45.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4008860 2021-03-09T10:03:45.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.476+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:45.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.476+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:45.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:45.476+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:50.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:03:50.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:50.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8827 cs=0 l=1 rev1=1 rx=0x7f21e40a6040 tx=0x7f21e4003b60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:50.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8827 cs=0 l=1 rev1=1 rx=0x7f21e40a6040 tx=0x7f21e4003b60).stop 2021-03-09T10:03:50.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:50.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=CLOSED pgs=8827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:50.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:50.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:50.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:50.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0276e0 con 0x7f21cc005e30 2021-03-09T10:03:50.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d00 con 0x7f21c4003cb0 2021-03-09T10:03:50.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022e60 con 0x7f21cc02f1a0 2021-03-09T10:03:50.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:50.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:50.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:50.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:50.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:50.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:50.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:50.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:50.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:50.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094fb0 con 0x7f21cc02f1a0 2021-03-09T10:03:50.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013ea0 con 0x7f21cc02f1a0 2021-03-09T10:03:50.481 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29967 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:50.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.472+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8839 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc004020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:50.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065570 con 0x7f21cc02f1a0 2021-03-09T10:03:50.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065740 con 0x7f21cc02f1a0 2021-03-09T10:03:50.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc02f1a0 2021-03-09T10:03:50.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.480+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:50.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:50.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:50.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:03:53.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:54.008 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:03:55.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:55.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:55.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8839 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc004020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:55.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8839 cs=0 l=1 rev1=1 rx=0x7f21dc0137f0 tx=0x7f21dc004020).stop 2021-03-09T10:03:55.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:55.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:55.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:55.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:55.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:55.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004c60 con 0x7f21cc005e30 2021-03-09T10:03:55.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047850 con 0x7f21c4003cb0 2021-03-09T10:03:55.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085ee0 con 0x7f21c4008860 2021-03-09T10:03:55.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:03:55.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:55.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:55.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:03:55.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:55.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:55.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:55.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:55.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:55.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086020 con 0x7f21c4008860 2021-03-09T10:03:55.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094fb0 con 0x7f21c4008860 2021-03-09T10:03:55.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8848 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:55.488 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:03:55.480+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30003 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:03:55.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4008860 2021-03-09T10:03:55.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4008860 2021-03-09T10:03:55.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21c4008860 2021-03-09T10:03:55.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.488+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:03:55.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:03:55.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:03:55.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:00.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:00.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:00.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8848 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:00.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 secure :-1 s=READY pgs=8848 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0b6d90).stop 2021-03-09T10:04:00.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 msgr2=0x7f21cc047360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:00.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4008860 0x7f21cc047360 unknown :-1 s=CLOSED pgs=8848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:00.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:00.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:00.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:00.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085ee0 con 0x7f21cc005e30 2021-03-09T10:04:00.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b6a0 con 0x7f21c4003cb0 2021-03-09T10:04:00.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095150 con 0x7f21cc02f1a0 2021-03-09T10:04:00.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:00.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:00.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:00.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:00.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:00.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:00.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:00.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:00.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:00.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013ea0 con 0x7f21cc02f1a0 2021-03-09T10:04:00.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086020 con 0x7f21cc02f1a0 2021-03-09T10:04:00.493 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30042 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:00.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8859 cs=0 l=1 rev1=1 rx=0x7f21dc0aca80 tx=0x7f21dc0b8e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:00.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc033110 con 0x7f21cc02f1a0 2021-03-09T10:04:00.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21cc02f1a0 2021-03-09T10:04:00.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc065200 con 0x7f21cc02f1a0 2021-03-09T10:04:00.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.492+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:00.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:00.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:00.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:05.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:05.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:05.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8859 cs=0 l=1 rev1=1 rx=0x7f21dc0aca80 tx=0x7f21dc0b8e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:05.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8859 cs=0 l=1 rev1=1 rx=0x7f21dc0aca80 tx=0x7f21dc0b8e90).stop 2021-03-09T10:04:05.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:05.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:05.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:05.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:05.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:05.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047740 con 0x7f21cc005e30 2021-03-09T10:04:05.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047880 con 0x7f21c4003cb0 2021-03-09T10:04:05.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084220 con 0x7f21cc047360 2021-03-09T10:04:05.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:05.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:05.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:05.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:05.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:05.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:05.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:05.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:05.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:05.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084360 con 0x7f21cc047360 2021-03-09T10:04:05.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.488+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013ea0 con 0x7f21cc047360 2021-03-09T10:04:05.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8869 cs=0 l=1 rev1=1 rx=0x7f21dc02b5d0 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:05.504 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:05.492+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30078 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:05.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc072a20 con 0x7f21cc047360 2021-03-09T10:04:05.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc072a20 con 0x7f21cc047360 2021-03-09T10:04:05.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc047360 2021-03-09T10:04:05.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.500+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:05.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:05.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:05.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:10.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:10.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:10.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8869 cs=0 l=1 rev1=1 rx=0x7f21dc02b5d0 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:10.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8869 cs=0 l=1 rev1=1 rx=0x7f21dc02b5d0 tx=0x7f21dc080000).stop 2021-03-09T10:04:10.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:10.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=CLOSED pgs=8869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:10.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:10.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:10.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:10.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084220 con 0x7f21cc005e30 2021-03-09T10:04:10.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f980 con 0x7f21c4003cb0 2021-03-09T10:04:10.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013ea0 con 0x7f21cc02f1a0 2021-03-09T10:04:10.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:10.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:10.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:10.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:10.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:10.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:10.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:10.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:10.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:10.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0844f0 con 0x7f21cc02f1a0 2021-03-09T10:04:10.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084360 con 0x7f21cc02f1a0 2021-03-09T10:04:10.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8878 cs=0 l=1 rev1=1 rx=0x7f21dc097560 tx=0x7f21dc081070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:10.504 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:10.496+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30117 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:10.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3770 con 0x7f21cc02f1a0 2021-03-09T10:04:10.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3590 con 0x7f21cc02f1a0 2021-03-09T10:04:10.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21cc02f1a0 2021-03-09T10:04:10.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.500+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:10.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:10.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:10.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:15.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:15.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:15.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8878 cs=0 l=1 rev1=1 rx=0x7f21dc097560 tx=0x7f21dc081070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:15.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 secure :-1 s=READY pgs=8878 cs=0 l=1 rev1=1 rx=0x7f21dc097560 tx=0x7f21dc081070).stop 2021-03-09T10:04:15.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 msgr2=0x7f21cc0138f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:15.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc02f1a0 0x7f21cc0138f0 unknown :-1 s=CLOSED pgs=8878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:15.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:15.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:15.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:15.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0477a0 con 0x7f21cc005e30 2021-03-09T10:04:15.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012720 con 0x7f21c4003cb0 2021-03-09T10:04:15.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015eb0 con 0x7f21cc047360 2021-03-09T10:04:15.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:15.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:15.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:15.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:15.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:15.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:15.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:15.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:15.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:15.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084220 con 0x7f21cc047360 2021-03-09T10:04:15.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00fb60 con 0x7f21cc047360 2021-03-09T10:04:15.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f21cc098630 tx=0x7f21e404aa40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:15.510 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:15.500+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30153 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:15.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404ac40 con 0x7f21cc047360 2021-03-09T10:04:15.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404ac40 con 0x7f21cc047360 2021-03-09T10:04:15.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404abd0 con 0x7f21cc047360 2021-03-09T10:04:15.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.508+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:15.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:15.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:15.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:20.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:20.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:20.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f21cc098630 tx=0x7f21e404aa40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:20.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f21cc098630 tx=0x7f21e404aa40).stop 2021-03-09T10:04:20.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:20.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=CLOSED pgs=8890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:20.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:20.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:20.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:20.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0477a0 con 0x7f21cc005e30 2021-03-09T10:04:20.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21c4003cb0 2021-03-09T10:04:20.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f4a0 con 0x7f21cc0138f0 2021-03-09T10:04:20.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:20.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:20.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:20.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:20.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:20.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:20.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:20.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:20.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:20.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084680 con 0x7f21cc0138f0 2021-03-09T10:04:20.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0844f0 con 0x7f21cc0138f0 2021-03-09T10:04:20.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.503+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02efb0 secure :-1 s=READY pgs=8899 cs=0 l=1 rev1=1 rx=0x7f21dc0b3b60 tx=0x7f21dc008e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:20.523 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:20.507+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30191 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:20.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0971f0 con 0x7f21cc0138f0 2021-03-09T10:04:20.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.507+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc090e90 con 0x7f21cc0138f0 2021-03-09T10:04:20.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21cc0138f0 2021-03-09T10:04:20.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.511+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:20.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.511+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:20.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:20.511+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:24.048 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:24.061 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:25.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:25.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:25.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02efb0 secure :-1 s=READY pgs=8899 cs=0 l=1 rev1=1 rx=0x7f21dc0b3b60 tx=0x7f21dc008e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:25.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02efb0 secure :-1 s=READY pgs=8899 cs=0 l=1 rev1=1 rx=0x7f21dc0b3b60 tx=0x7f21dc008e90).stop 2021-03-09T10:04:25.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02efb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:25.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02efb0 unknown :-1 s=CLOSED pgs=8899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:25.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:25.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:25.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:25.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0477a0 con 0x7f21cc005e30 2021-03-09T10:04:25.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056a80 con 0x7f21c4003cb0 2021-03-09T10:04:25.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21cc047360 2021-03-09T10:04:25.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:25.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:25.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:25.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:25.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:25.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:25.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:25.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:25.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:25.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0844f0 con 0x7f21cc047360 2021-03-09T10:04:25.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084220 con 0x7f21cc047360 2021-03-09T10:04:25.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8911 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e409ce80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:25.520 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:25.511+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30228 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:25.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.515+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4050020 con 0x7f21cc047360 2021-03-09T10:04:25.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc047360 2021-03-09T10:04:25.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc047360 2021-03-09T10:04:25.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.519+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:25.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:25.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:25.519+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:30.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.515+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:30.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:30.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8911 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e409ce80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:30.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8911 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e409ce80).stop 2021-03-09T10:04:30.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:30.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=CLOSED pgs=8911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:30.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:30.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:30.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:30.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e2b0 con 0x7f21cc005e30 2021-03-09T10:04:30.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047780 con 0x7f21c4003cb0 2021-03-09T10:04:30.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084220 con 0x7f21cc0138f0 2021-03-09T10:04:30.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:30.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:30.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:30.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:30.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:30.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:30.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:30.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:30.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:30.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:04:30.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.519+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0844f0 con 0x7f21cc0138f0 2021-03-09T10:04:30.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.523+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 secure :-1 s=READY pgs=8920 cs=0 l=1 rev1=1 rx=0x7f21e40a5490 tx=0x7f21e4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:30.526 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:30.523+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30266 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:30.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a3040 con 0x7f21cc0138f0 2021-03-09T10:04:30.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404abd0 con 0x7f21cc0138f0 2021-03-09T10:04:30.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3040 con 0x7f21cc0138f0 2021-03-09T10:04:30.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.535+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:30.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.535+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:30.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:30.535+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:35.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:35.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:35.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 secure :-1 s=READY pgs=8920 cs=0 l=1 rev1=1 rx=0x7f21e40a5490 tx=0x7f21e4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:35.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 secure :-1 s=READY pgs=8920 cs=0 l=1 rev1=1 rx=0x7f21e40a5490 tx=0x7f21e4005180).stop 2021-03-09T10:04:35.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ef60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:35.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 unknown :-1 s=CLOSED pgs=8920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:35.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:35.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:35.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:35.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047780 con 0x7f21cc005e30 2021-03-09T10:04:35.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0844f0 con 0x7f21c4003cb0 2021-03-09T10:04:35.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e2b0 con 0x7f21cc047360 2021-03-09T10:04:35.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:35.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:35.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:35.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:35.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.523+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:35.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:35.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:35.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:35.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:35.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084220 con 0x7f21cc047360 2021-03-09T10:04:35.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084680 con 0x7f21cc047360 2021-03-09T10:04:35.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8931 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc029f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:35.533 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30303 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:35.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc047360 2021-03-09T10:04:35.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc047360 2021-03-09T10:04:35.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21cc047360 2021-03-09T10:04:35.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.535+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:35.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.535+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:35.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:35.535+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:40.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:40.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:40.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.527+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8931 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc029f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:40.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8931 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc029f10).stop 2021-03-09T10:04:40.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:40.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=CLOSED pgs=8931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:40.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:40.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:40.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:40.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04bdc0 con 0x7f21cc005e30 2021-03-09T10:04:40.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02eb60 con 0x7f21c4003cb0 2021-03-09T10:04:40.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f450 con 0x7f21cc0138f0 2021-03-09T10:04:40.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:40.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:40.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:40.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:40.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:40.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:40.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:40.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:40.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:40.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0844f0 con 0x7f21cc0138f0 2021-03-09T10:04:40.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:04:40.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 secure :-1 s=READY pgs=8942 cs=0 l=1 rev1=1 rx=0x7f21cc052e60 tx=0x7f21e404ae70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:40.539 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:40.535+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30341 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:40.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc0138f0 2021-03-09T10:04:40.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40241e0 con 0x7f21cc0138f0 2021-03-09T10:04:40.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.539+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:04:40.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.539+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:40.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.539+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:40.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:40.539+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:45.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:45.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:45.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 secure :-1 s=READY pgs=8942 cs=0 l=1 rev1=1 rx=0x7f21cc052e60 tx=0x7f21e404ae70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:45.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 secure :-1 s=READY pgs=8942 cs=0 l=1 rev1=1 rx=0x7f21cc052e60 tx=0x7f21e404ae70).stop 2021-03-09T10:04:45.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ef60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:45.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ef60 unknown :-1 s=CLOSED pgs=8942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:45.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:45.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:45.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:45.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f450 con 0x7f21cc005e30 2021-03-09T10:04:45.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21c4003cb0 2021-03-09T10:04:45.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040070 con 0x7f21cc047360 2021-03-09T10:04:45.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:45.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:45.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:45.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:45.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:45.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:45.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.535+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:45.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:45.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:45.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036150 con 0x7f21cc047360 2021-03-09T10:04:45.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084220 con 0x7f21cc047360 2021-03-09T10:04:45.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8952 cs=0 l=1 rev1=1 rx=0x7f21dc0abc60 tx=0x7f21dc0abf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:45.544 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:45.539+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30378 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:45.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ab8c0 con 0x7f21cc047360 2021-03-09T10:04:45.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc047360 2021-03-09T10:04:45.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.547+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc047360 2021-03-09T10:04:45.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.547+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:45.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.547+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:45.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:45.547+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:50.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:50.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:50.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8952 cs=0 l=1 rev1=1 rx=0x7f21dc0abc60 tx=0x7f21dc0abf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:50.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 secure :-1 s=READY pgs=8952 cs=0 l=1 rev1=1 rx=0x7f21dc0abc60 tx=0x7f21dc0abf70).stop 2021-03-09T10:04:50.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.539+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc083d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:50.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.539+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc083d30 unknown :-1 s=CLOSED pgs=8952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:50.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:50.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc056590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:50.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:50.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084220 con 0x7f21cc005e30 2021-03-09T10:04:50.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040210 con 0x7f21c4003cb0 2021-03-09T10:04:50.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:04:50.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:50.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:50.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:50.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:50.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc056590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:50.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:50.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:50.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:50.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:50.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056a80 con 0x7f21cc0138f0 2021-03-09T10:04:50.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.543+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0844f0 con 0x7f21cc0138f0 2021-03-09T10:04:50.550 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:50.547+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30416 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:50.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc056590 secure :-1 s=READY pgs=8969 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4016ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:50.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404aca0 con 0x7f21cc0138f0 2021-03-09T10:04:50.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.555+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404aca0 con 0x7f21cc0138f0 2021-03-09T10:04:50.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.555+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:04:50.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.555+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:50.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.555+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:50.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:50.555+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:04:54.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:54.112 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:04:55.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc056590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:55.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc056590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:55.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc056590 secure :-1 s=READY pgs=8969 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4016ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:55.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc056590 secure :-1 s=READY pgs=8969 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4016ab0).stop 2021-03-09T10:04:55.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc056590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:55.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc056590 unknown :-1 s=CLOSED pgs=8969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:55.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:55.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:55.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:55.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d220 con 0x7f21cc005e30 2021-03-09T10:04:55.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21c4003cb0 2021-03-09T10:04:55.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0403b0 con 0x7f21cc047360 2021-03-09T10:04:55.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:04:55.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:55.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:04:55.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:55.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.547+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:55.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:55.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:55.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:55.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:55.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022130 con 0x7f21cc047360 2021-03-09T10:04:55.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036150 con 0x7f21cc047360 2021-03-09T10:04:55.557 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30453 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:04:55.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.551+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=8975 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:55.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.555+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc047360 2021-03-09T10:04:55.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.555+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc047360 2021-03-09T10:04:55.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21cc047360 2021-03-09T10:04:55.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.559+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:04:55.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.559+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:04:55.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:04:55.559+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:00.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:00.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:00.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=8975 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:00.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=8975 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc07ff90).stop 2021-03-09T10:05:00.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:00.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=CLOSED pgs=8975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:00.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:00.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:00.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:00.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046630 con 0x7f21cc005e30 2021-03-09T10:05:00.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d220 con 0x7f21c4003cb0 2021-03-09T10:05:00.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:05:00.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:00.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.551+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:00.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:00.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:00.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:00.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:00.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:00.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:00.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:00.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022130 con 0x7f21cc0138f0 2021-03-09T10:05:00.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056a80 con 0x7f21cc0138f0 2021-03-09T10:05:00.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.555+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 secure :-1 s=READY pgs=8984 cs=0 l=1 rev1=1 rx=0x7f21e4027c70 tx=0x7f21e408f100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:00.562 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:00.559+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30491 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:00.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21cc0138f0 2021-03-09T10:05:00.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21cc0138f0 2021-03-09T10:05:00.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:05:00.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.563+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:00.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.563+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:00.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:00.563+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:05.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:05.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:05.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 secure :-1 s=READY pgs=8984 cs=0 l=1 rev1=1 rx=0x7f21e4027c70 tx=0x7f21e408f100).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:05.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 secure :-1 s=READY pgs=8984 cs=0 l=1 rev1=1 rx=0x7f21e4027c70 tx=0x7f21e408f100).stop 2021-03-09T10:05:05.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:05.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 unknown :-1 s=CLOSED pgs=8984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:05.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:05.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:05.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:05.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046630 con 0x7f21cc005e30 2021-03-09T10:05:05.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21c4003cb0 2021-03-09T10:05:05.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022130 con 0x7f21cc047360 2021-03-09T10:05:05.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:05.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:05.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:05.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:05.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:05.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:05.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:05.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:05.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:05.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015e10 con 0x7f21cc047360 2021-03-09T10:05:05.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036150 con 0x7f21cc047360 2021-03-09T10:05:05.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0564d0 secure :-1 s=READY pgs=8995 cs=0 l=1 rev1=1 rx=0x7f21cc03dda0 tx=0x7f21dc0b6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:05.568 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:05.563+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30528 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:05.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc047360 2021-03-09T10:05:05.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0abcd0 con 0x7f21cc047360 2021-03-09T10:05:05.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc047360 2021-03-09T10:05:05.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.567+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:05.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.571+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:05.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:05.571+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:10.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:05:10.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:10.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0564d0 secure :-1 s=READY pgs=8995 cs=0 l=1 rev1=1 rx=0x7f21cc03dda0 tx=0x7f21dc0b6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:10.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0564d0 secure :-1 s=READY pgs=8995 cs=0 l=1 rev1=1 rx=0x7f21cc03dda0 tx=0x7f21dc0b6190).stop 2021-03-09T10:05:10.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0564d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:10.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0564d0 unknown :-1 s=CLOSED pgs=8995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:10.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:10.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:10.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:10.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082e00 con 0x7f21cc005e30 2021-03-09T10:05:10.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046630 con 0x7f21c4003cb0 2021-03-09T10:05:10.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:05:10.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:10.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:10.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:10.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:10.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:10.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:10.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:10.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:10.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:10.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015e10 con 0x7f21cc0138f0 2021-03-09T10:05:10.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036150 con 0x7f21cc0138f0 2021-03-09T10:05:10.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.567+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 secure :-1 s=READY pgs=9006 cs=0 l=1 rev1=1 rx=0x7f21dc02b000 tx=0x7f21dc062d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:10.574 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:10.571+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:10.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06a5b0 con 0x7f21cc0138f0 2021-03-09T10:05:10.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6320 con 0x7f21cc0138f0 2021-03-09T10:05:10.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:05:10.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.575+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:10.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.575+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:10.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:10.575+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:15.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:05:15.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:15.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 secure :-1 s=READY pgs=9006 cs=0 l=1 rev1=1 rx=0x7f21dc02b000 tx=0x7f21dc062d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:15.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 secure :-1 s=READY pgs=9006 cs=0 l=1 rev1=1 rx=0x7f21dc02b000 tx=0x7f21dc062d20).stop 2021-03-09T10:05:15.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:15.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040070 unknown :-1 s=CLOSED pgs=9006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:15.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082e00 con 0x7f21cc005e30 2021-03-09T10:05:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c80 con 0x7f21c4003cb0 2021-03-09T10:05:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046630 con 0x7f21cc047360 2021-03-09T10:05:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:15.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:15.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.570+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:15.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:15.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:15.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:15.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:15.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011b70 con 0x7f21cc047360 2021-03-09T10:05:15.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036150 con 0x7f21cc047360 2021-03-09T10:05:15.579 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30603 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:15.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.574+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=9022 cs=0 l=1 rev1=1 rx=0x7f21dc012000 tx=0x7f21dc0b6360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:15.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.578+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:05:15.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.578+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3eb0 con 0x7f21cc047360 2021-03-09T10:05:15.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.582+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ba000 con 0x7f21cc047360 2021-03-09T10:05:15.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.582+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:15.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.582+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:15.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:15.582+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:20.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:20.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:20.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=9022 cs=0 l=1 rev1=1 rx=0x7f21dc012000 tx=0x7f21dc0b6360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:20.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=9022 cs=0 l=1 rev1=1 rx=0x7f21dc012000 tx=0x7f21dc0b6360).stop 2021-03-09T10:05:20.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:20.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=CLOSED pgs=9022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:20.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:20.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:20.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:20.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082e00 con 0x7f21cc005e30 2021-03-09T10:05:20.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c80 con 0x7f21c4003cb0 2021-03-09T10:05:20.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dbd0 con 0x7f21cc0138f0 2021-03-09T10:05:20.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:20.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:20.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:20.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.574+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:20.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:20.584 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30641 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:20.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:20.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:20.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:20.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:20.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040070 con 0x7f21cc0138f0 2021-03-09T10:05:20.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.578+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:05:20.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.582+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055ff0 secure :-1 s=READY pgs=9031 cs=0 l=1 rev1=1 rx=0x7f21dc00e000 tx=0x7f21dc0572b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:20.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.582+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052c30 con 0x7f21cc0138f0 2021-03-09T10:05:20.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc0138f0 2021-03-09T10:05:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029d20 con 0x7f21cc0138f0 2021-03-09T10:05:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.586+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.590+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:20.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:20.590+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:24.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:24.203 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:25.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:25.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:25.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055ff0 secure :-1 s=READY pgs=9031 cs=0 l=1 rev1=1 rx=0x7f21dc00e000 tx=0x7f21dc0572b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:25.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055ff0 secure :-1 s=READY pgs=9031 cs=0 l=1 rev1=1 rx=0x7f21dc00e000 tx=0x7f21dc0572b0).stop 2021-03-09T10:05:25.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:25.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055ff0 unknown :-1 s=CLOSED pgs=9031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:25.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:25.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:25.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:25.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c80 con 0x7f21cc005e30 2021-03-09T10:05:25.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0358c0 con 0x7f21c4003cb0 2021-03-09T10:05:25.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21cc047360 2021-03-09T10:05:25.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:25.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:25.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:25.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:25.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:25.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:25.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:25.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:25.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:25.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01dbd0 con 0x7f21cc047360 2021-03-09T10:05:25.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056a80 con 0x7f21cc047360 2021-03-09T10:05:25.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.582+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=9038 cs=0 l=1 rev1=1 rx=0x7f21e4008300 tx=0x7f21e4008230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:25.591 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:25.586+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30678 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:25.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc047360 2021-03-09T10:05:25.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc047360 2021-03-09T10:05:25.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.590+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc047360 2021-03-09T10:05:25.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.590+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:25.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.590+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:25.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:25.590+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:30.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:30.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:30.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=9038 cs=0 l=1 rev1=1 rx=0x7f21e4008300 tx=0x7f21e4008230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:30.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 secure :-1 s=READY pgs=9038 cs=0 l=1 rev1=1 rx=0x7f21e4008300 tx=0x7f21e4008230).stop 2021-03-09T10:05:30.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc0844f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:30.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc0844f0 unknown :-1 s=CLOSED pgs=9038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:30.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:30.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:30.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:30.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045610 con 0x7f21cc005e30 2021-03-09T10:05:30.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c80 con 0x7f21c4003cb0 2021-03-09T10:05:30.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:05:30.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:30.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:30.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:30.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:30.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.586+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:30.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:30.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:30.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:30.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:30.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026ab0 con 0x7f21cc0138f0 2021-03-09T10:05:30.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040070 con 0x7f21cc0138f0 2021-03-09T10:05:30.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055f50 secure :-1 s=READY pgs=9049 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc053040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:30.596 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:30.590+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30716 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:30.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.594+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc09b220 con 0x7f21cc0138f0 2021-03-09T10:05:30.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.594+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065a30 con 0x7f21cc0138f0 2021-03-09T10:05:30.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc0138f0 2021-03-09T10:05:30.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.598+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:30.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.598+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:30.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:30.598+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:35.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:35.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:35.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055f50 secure :-1 s=READY pgs=9049 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc053040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:35.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055f50 secure :-1 s=READY pgs=9049 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc053040).stop 2021-03-09T10:05:35.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:35.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055f50 unknown :-1 s=CLOSED pgs=9049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:35.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:35.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:35.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:35.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e650 con 0x7f21cc005e30 2021-03-09T10:05:35.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045610 con 0x7f21c4003cb0 2021-03-09T10:05:35.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b4f0 con 0x7f21cc047360 2021-03-09T10:05:35.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:35.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:35.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:35.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.590+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:35.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:35.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:35.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:35.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:35.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:35.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011b70 con 0x7f21cc047360 2021-03-09T10:05:35.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01dbd0 con 0x7f21cc047360 2021-03-09T10:05:35.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.594+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 secure :-1 s=READY pgs=9059 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e40092a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:35.602 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:35.598+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30753 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:35.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409cc80 con 0x7f21cc047360 2021-03-09T10:05:35.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4008330 con 0x7f21cc047360 2021-03-09T10:05:35.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cc80 con 0x7f21cc047360 2021-03-09T10:05:35.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.598+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:35.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.602+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:35.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:35.602+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:40.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:40.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:40.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 secure :-1 s=READY pgs=9059 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e40092a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:40.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 secure :-1 s=READY pgs=9059 cs=0 l=1 rev1=1 rx=0x7f21e407f560 tx=0x7f21e40092a0).stop 2021-03-09T10:05:40.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:40.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 unknown :-1 s=CLOSED pgs=9059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:40.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:40.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:40.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:40.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0130b0 con 0x7f21cc005e30 2021-03-09T10:05:40.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e4b0 con 0x7f21c4003cb0 2021-03-09T10:05:40.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e5f0 con 0x7f21cc0138f0 2021-03-09T10:05:40.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:40.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:40.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:40.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:40.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:40.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:40.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:40.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:40.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:40.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056100 con 0x7f21cc0138f0 2021-03-09T10:05:40.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011b70 con 0x7f21cc0138f0 2021-03-09T10:05:40.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.602+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055c10 secure :-1 s=READY pgs=9070 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4028d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:40.608 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:40.602+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30791 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:40.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.602+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e405e020 con 0x7f21cc0138f0 2021-03-09T10:05:40.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:05:40.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e405e020 con 0x7f21cc0138f0 2021-03-09T10:05:40.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.610+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:40.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.610+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:40.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:40.610+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:45.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:45.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:45.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055c10 secure :-1 s=READY pgs=9070 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4028d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:45.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055c10 secure :-1 s=READY pgs=9070 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4028d20).stop 2021-03-09T10:05:45.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:45.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055c10 unknown :-1 s=CLOSED pgs=9070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:45.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:45.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:45.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:45.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015e10 con 0x7f21cc005e30 2021-03-09T10:05:45.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017600 con 0x7f21c4003cb0 2021-03-09T10:05:45.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017740 con 0x7f21cc047360 2021-03-09T10:05:45.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:45.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:45.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:45.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:45.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:45.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:45.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:45.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:45.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:45.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e4b0 con 0x7f21cc047360 2021-03-09T10:05:45.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056100 con 0x7f21cc047360 2021-03-09T10:05:45.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 secure :-1 s=READY pgs=9080 cs=0 l=1 rev1=1 rx=0x7f21e4027c70 tx=0x7f21e404ed40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:45.614 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30828 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:45.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007160 con 0x7f21cc047360 2021-03-09T10:05:45.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.606+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40072c0 con 0x7f21cc047360 2021-03-09T10:05:45.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007160 con 0x7f21cc047360 2021-03-09T10:05:45.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.610+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:45.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.614+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:45.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:45.614+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:50.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:05:50.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:50.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 secure :-1 s=READY pgs=9080 cs=0 l=1 rev1=1 rx=0x7f21e4027c70 tx=0x7f21e404ed40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:50.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 secure :-1 s=READY pgs=9080 cs=0 l=1 rev1=1 rx=0x7f21e4027c70 tx=0x7f21e404ed40).stop 2021-03-09T10:05:50.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:50.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026c40 unknown :-1 s=CLOSED pgs=9080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:50.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:50.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:50.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:50.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e640 con 0x7f21cc005e30 2021-03-09T10:05:50.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f60 con 0x7f21c4003cb0 2021-03-09T10:05:50.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0560a0 con 0x7f21cc0138f0 2021-03-09T10:05:50.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:50.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:50.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:50.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:50.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:50.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:50.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:50.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:50.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:50.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0844f0 con 0x7f21cc0138f0 2021-03-09T10:05:50.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026ab0 con 0x7f21cc0138f0 2021-03-09T10:05:50.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9091 cs=0 l=1 rev1=1 rx=0x7f21dc0800c0 tx=0x7f21dc062f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:50.619 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:50.614+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30866 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:50.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065a30 con 0x7f21cc0138f0 2021-03-09T10:05:50.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065a30 con 0x7f21cc0138f0 2021-03-09T10:05:50.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc0138f0 2021-03-09T10:05:50.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.622+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:50.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.622+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:50.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:50.622+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:05:54.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:54.249 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:05:55.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:55.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:55.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9091 cs=0 l=1 rev1=1 rx=0x7f21dc0800c0 tx=0x7f21dc062f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:55.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9091 cs=0 l=1 rev1=1 rx=0x7f21dc0800c0 tx=0x7f21dc062f80).stop 2021-03-09T10:05:55.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:55.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:55.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:55.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:55.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:55.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e640 con 0x7f21cc005e30 2021-03-09T10:05:55.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc00 con 0x7f21c4003cb0 2021-03-09T10:05:55.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04abc0 con 0x7f21cc047360 2021-03-09T10:05:55.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:05:55.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:55.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:05:55.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:55.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:55.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:55.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:55.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:55.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:55.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026fa0 con 0x7f21cc047360 2021-03-09T10:05:55.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0844f0 con 0x7f21cc047360 2021-03-09T10:05:55.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026ab0 secure :-1 s=READY pgs=9104 cs=0 l=1 rev1=1 rx=0x7f21dc057280 tx=0x7f21dc0b3220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:55.624 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:05:55.618+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30903 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:05:55.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065e90 con 0x7f21cc047360 2021-03-09T10:05:55.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc065e90 con 0x7f21cc047360 2021-03-09T10:05:55.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0a0cf0 con 0x7f21cc047360 2021-03-09T10:05:55.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.626+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:05:55.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:05:55.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:05:55.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:00.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:06:00.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:00.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026ab0 secure :-1 s=READY pgs=9104 cs=0 l=1 rev1=1 rx=0x7f21dc057280 tx=0x7f21dc0b3220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:00.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026ab0 secure :-1 s=READY pgs=9104 cs=0 l=1 rev1=1 rx=0x7f21dc057280 tx=0x7f21dc0b3220).stop 2021-03-09T10:06:00.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:00.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026ab0 unknown :-1 s=CLOSED pgs=9104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:00.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:00.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:00.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:00.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e640 con 0x7f21cc005e30 2021-03-09T10:06:00.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ab70 con 0x7f21c4003cb0 2021-03-09T10:06:00.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055360 con 0x7f21cc0138f0 2021-03-09T10:06:00.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:00.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:00.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:00.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:00.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:00.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:00.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:00.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:00.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:00.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:06:00.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.622+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026fa0 con 0x7f21cc0138f0 2021-03-09T10:06:00.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9113 cs=0 l=1 rev1=1 rx=0x7f21cc027f00 tx=0x7f21dc06a0e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:00.631 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:00.626+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30941 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:00.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc0138f0 2021-03-09T10:06:00.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0a0ea0 con 0x7f21cc0138f0 2021-03-09T10:06:00.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:06:00.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.630+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:00.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:00.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:00.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:05.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:05.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:05.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9113 cs=0 l=1 rev1=1 rx=0x7f21cc027f00 tx=0x7f21dc06a0e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:05.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9113 cs=0 l=1 rev1=1 rx=0x7f21cc027f00 tx=0x7f21dc06a0e0).stop 2021-03-09T10:06:05.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:05.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:05.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:05.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:05.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:05.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012750 con 0x7f21cc005e30 2021-03-09T10:06:05.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:06:05.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015e10 con 0x7f21cc047360 2021-03-09T10:06:05.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:05.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:05.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:05.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:05.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:05.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:05.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:05.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:05.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:05.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc055360 con 0x7f21cc047360 2021-03-09T10:06:05.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e4b0 con 0x7f21cc047360 2021-03-09T10:06:05.636 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30978 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:05.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.630+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9125 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e407f290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:05.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404ee80 con 0x7f21cc047360 2021-03-09T10:06:05.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404ee80 con 0x7f21cc047360 2021-03-09T10:06:05.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc047360 2021-03-09T10:06:05.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.638+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:05.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.638+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:05.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:05.638+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:10.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:10.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:10.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9125 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e407f290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:10.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9125 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e407f290).stop 2021-03-09T10:06:10.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:10.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:10.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:10.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:10.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:10.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a450 con 0x7f21cc005e30 2021-03-09T10:06:10.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0554f0 con 0x7f21c4003cb0 2021-03-09T10:06:10.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055630 con 0x7f21cc0138f0 2021-03-09T10:06:10.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:10.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:10.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:10.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:10.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:10.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:10.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:10.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:10.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:10.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041b20 con 0x7f21cc0138f0 2021-03-09T10:06:10.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:06:10.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.633+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9134 cs=0 l=1 rev1=1 rx=0x7f21dc0800c0 tx=0x7f21dc0beff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:10.642 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:10.637+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31016 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:10.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc0138f0 2021-03-09T10:06:10.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc0138f0 2021-03-09T10:06:10.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.641+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0a0dd0 con 0x7f21cc0138f0 2021-03-09T10:06:10.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.641+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:10.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.641+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:10.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:10.641+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:15.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:15.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:15.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9134 cs=0 l=1 rev1=1 rx=0x7f21dc0800c0 tx=0x7f21dc0beff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:15.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9134 cs=0 l=1 rev1=1 rx=0x7f21dc0800c0 tx=0x7f21dc0beff0).stop 2021-03-09T10:06:15.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:15.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:15.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:15.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:15.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:15.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055690 con 0x7f21cc005e30 2021-03-09T10:06:15.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:06:15.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a2b0 con 0x7f21cc047360 2021-03-09T10:06:15.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:15.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:15.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:15.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:15.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:15.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:15.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:15.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:15.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:15.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a3f0 con 0x7f21cc047360 2021-03-09T10:06:15.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041b20 con 0x7f21cc047360 2021-03-09T10:06:15.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.641+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9144 cs=0 l=1 rev1=1 rx=0x7f21dc0333f0 tx=0x7f21dc083980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:15.649 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:15.641+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31052 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:15.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.641+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:06:15.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:06:15.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0a0dd0 con 0x7f21cc047360 2021-03-09T10:06:15.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.649+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:15.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.649+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:15.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:15.649+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:20.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:06:20.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:20.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9144 cs=0 l=1 rev1=1 rx=0x7f21dc0333f0 tx=0x7f21dc083980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:20.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9144 cs=0 l=1 rev1=1 rx=0x7f21dc0333f0 tx=0x7f21dc083980).stop 2021-03-09T10:06:20.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:20.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:20.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:20.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:20.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:20.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:20.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:20.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a2b0 con 0x7f21cc005e30 2021-03-09T10:06:20.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e070 con 0x7f21c4003cb0 2021-03-09T10:06:20.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041310 con 0x7f21cc0138f0 2021-03-09T10:06:20.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:20.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:20.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:20.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:20.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:20.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:20.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:20.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:06:20.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.645+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a3f0 con 0x7f21cc0138f0 2021-03-09T10:06:20.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9156 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:20.657 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:20.649+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31091 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:20.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083e80 con 0x7f21cc0138f0 2021-03-09T10:06:20.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:06:20.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ac1e0 con 0x7f21cc0138f0 2021-03-09T10:06:20.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.653+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:20.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:20.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:20.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:24.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:24.322 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:25.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:25.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:25.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9156 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:25.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9156 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc010d20).stop 2021-03-09T10:06:25.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:25.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:25.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:25.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:25.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:25.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03ebc0 con 0x7f21cc005e30 2021-03-09T10:06:25.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f7e0 con 0x7f21c4003cb0 2021-03-09T10:06:25.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e070 con 0x7f21cc047360 2021-03-09T10:06:25.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:25.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:25.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:25.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:25.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.649+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:25.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:25.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:25.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:25.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:25.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc055f60 con 0x7f21cc047360 2021-03-09T10:06:25.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a3f0 con 0x7f21cc047360 2021-03-09T10:06:25.660 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:25.653+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31127 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:25.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9170 cs=0 l=1 rev1=1 rx=0x7f21dc087490 tx=0x7f21dc07ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:25.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083e80 con 0x7f21cc047360 2021-03-09T10:06:25.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:06:25.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc047360 2021-03-09T10:06:25.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.661+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:25.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.661+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:25.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:25.661+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:30.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:30.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:30.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9170 cs=0 l=1 rev1=1 rx=0x7f21dc087490 tx=0x7f21dc07ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:30.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9170 cs=0 l=1 rev1=1 rx=0x7f21dc087490 tx=0x7f21dc07ff90).stop 2021-03-09T10:06:30.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:30.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:30.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:30.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:30.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:30.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f60 con 0x7f21cc005e30 2021-03-09T10:06:30.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e780 con 0x7f21c4003cb0 2021-03-09T10:06:30.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02e8c0 con 0x7f21cc0138f0 2021-03-09T10:06:30.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:30.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:30.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:30.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:30.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:30.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:30.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:30.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:30.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:30.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ea00 con 0x7f21cc0138f0 2021-03-09T10:06:30.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.657+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc055360 con 0x7f21cc0138f0 2021-03-09T10:06:30.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.661+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9180 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e407d1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:30.666 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:30.661+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31166 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:30.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.661+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc0138f0 2021-03-09T10:06:30.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:06:30.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc0138f0 2021-03-09T10:06:30.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.669+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:30.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.669+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:30.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:30.669+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:35.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:35.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:35.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9180 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e407d1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:35.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9180 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e407d1d0).stop 2021-03-09T10:06:35.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:35.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:35.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:35.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:35.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:35.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055500 con 0x7f21cc005e30 2021-03-09T10:06:35.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:06:35.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0844f0 con 0x7f21cc047360 2021-03-09T10:06:35.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:35.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:35.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:35.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:35.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:35.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:35.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:35.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:35.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:35.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084630 con 0x7f21cc047360 2021-03-09T10:06:35.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a3f0 con 0x7f21cc047360 2021-03-09T10:06:35.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9187 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:35.673 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:35.665+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31202 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:35.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc029350 con 0x7f21cc047360 2021-03-09T10:06:35.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc047360 2021-03-09T10:06:35.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc047360 2021-03-09T10:06:35.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.673+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:35.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.673+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:35.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:35.673+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:40.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:40.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:40.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9187 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:40.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9187 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc077e50).stop 2021-03-09T10:06:40.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:40.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:40.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:40.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:40.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:40.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0844f0 con 0x7f21cc005e30 2021-03-09T10:06:40.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027e40 con 0x7f21c4003cb0 2021-03-09T10:06:40.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a3f0 con 0x7f21cc0138f0 2021-03-09T10:06:40.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:40.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:40.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:40.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:40.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.669+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:40.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:40.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:40.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:40.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:40.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:06:40.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084630 con 0x7f21cc0138f0 2021-03-09T10:06:40.679 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31241 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:40.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9200 cs=0 l=1 rev1=1 rx=0x7f21dc02b570 tx=0x7f21dc0a0cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:40.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b6d80 con 0x7f21cc0138f0 2021-03-09T10:06:40.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.677+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc0138f0 2021-03-09T10:06:40.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.681+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc0138f0 2021-03-09T10:06:40.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.681+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:40.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.681+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:40.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:40.681+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:45.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:45.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:45.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9200 cs=0 l=1 rev1=1 rx=0x7f21dc02b570 tx=0x7f21dc0a0cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:45.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9200 cs=0 l=1 rev1=1 rx=0x7f21dc02b570 tx=0x7f21dc0a0cb0).stop 2021-03-09T10:06:45.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:45.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:45.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:45.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:45.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:45.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083f60 con 0x7f21cc005e30 2021-03-09T10:06:45.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f940 con 0x7f21c4003cb0 2021-03-09T10:06:45.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ce60 con 0x7f21cc047360 2021-03-09T10:06:45.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:45.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:45.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:45.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:45.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.673+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:45.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:45.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:45.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:45.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:45.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a3f0 con 0x7f21cc047360 2021-03-09T10:06:45.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084630 con 0x7f21cc047360 2021-03-09T10:06:45.684 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31277 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:45.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.677+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9211 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc080000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:45.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.681+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0babe0 con 0x7f21cc047360 2021-03-09T10:06:45.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0baa00 con 0x7f21cc047360 2021-03-09T10:06:45.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.689+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b6d80 con 0x7f21cc047360 2021-03-09T10:06:45.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.689+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:45.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.689+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:45.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:45.689+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:50.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:50.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:50.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9211 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc080000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:50.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9211 cs=0 l=1 rev1=1 rx=0x7f21dc0841d0 tx=0x7f21dc080000).stop 2021-03-09T10:06:50.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:50.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:50.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:50.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:50.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:50.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ce60 con 0x7f21cc005e30 2021-03-09T10:06:50.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f60 con 0x7f21c4003cb0 2021-03-09T10:06:50.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030840 con 0x7f21cc0138f0 2021-03-09T10:06:50.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:50.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:50.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.677+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:50.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:50.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:50.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:50.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:50.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:50.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:50.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc027df0 con 0x7f21cc0138f0 2021-03-09T10:06:50.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a3f0 con 0x7f21cc0138f0 2021-03-09T10:06:50.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.681+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9219 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc032b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:50.690 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:50.685+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31316 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:50.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ab2b0 con 0x7f21cc0138f0 2021-03-09T10:06:50.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc0138f0 2021-03-09T10:06:50.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.689+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b6d80 con 0x7f21cc0138f0 2021-03-09T10:06:50.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.689+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:50.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.689+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:50.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:50.689+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:06:54.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:54.378 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:06:55.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:55.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:55.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9219 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc032b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:55.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9219 cs=0 l=1 rev1=1 rx=0x7f21dc073010 tx=0x7f21dc032b70).stop 2021-03-09T10:06:55.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:55.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:55.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:55.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:55.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:55.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ce60 con 0x7f21cc005e30 2021-03-09T10:06:55.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a3f0 con 0x7f21c4003cb0 2021-03-09T10:06:55.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:06:55.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:06:55.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:55.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:55.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:06:55.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:55.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:55.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:55.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:55.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:55.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084530 con 0x7f21cc047360 2021-03-09T10:06:55.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ea00 con 0x7f21cc047360 2021-03-09T10:06:55.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.689+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9230 cs=0 l=1 rev1=1 rx=0x7f21cc054e60 tx=0x7f21e4016ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:55.695 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:06:55.689+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31352 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:06:55.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc047360 2021-03-09T10:06:55.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc047360 2021-03-09T10:06:55.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.697+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc047360 2021-03-09T10:06:55.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.697+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:06:55.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.697+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:06:55.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:06:55.697+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:00.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:00.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:00.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9230 cs=0 l=1 rev1=1 rx=0x7f21cc054e60 tx=0x7f21e4016ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:00.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9230 cs=0 l=1 rev1=1 rx=0x7f21cc054e60 tx=0x7f21e4016ab0).stop 2021-03-09T10:07:00.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:00.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:00.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:00.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:00.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:00.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ea00 con 0x7f21cc005e30 2021-03-09T10:07:00.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084250 con 0x7f21c4003cb0 2021-03-09T10:07:00.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:07:00.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:00.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:00.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:00.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:00.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:00.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:00.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:00.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:00.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:00.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057450 con 0x7f21cc0138f0 2021-03-09T10:07:00.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc027df0 con 0x7f21cc0138f0 2021-03-09T10:07:00.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9240 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0a0a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:00.702 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:00.696+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31391 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:00.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21cc0138f0 2021-03-09T10:07:00.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.704+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09bdb0 con 0x7f21cc0138f0 2021-03-09T10:07:00.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.704+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc0138f0 2021-03-09T10:07:00.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.704+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:00.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.704+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:00.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:00.704+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:05.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:05.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:05.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9240 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0a0a40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:05.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9240 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc0a0a40).stop 2021-03-09T10:07:05.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:05.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:05.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:05.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:05.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:05.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:05.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ea00 con 0x7f21c4003cb0 2021-03-09T10:07:05.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:07:05.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:05.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:05.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:05.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:05.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.696+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:05.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:05.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:05.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:05.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:05.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc004c10 con 0x7f21cc047360 2021-03-09T10:07:05.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057450 con 0x7f21cc047360 2021-03-09T10:07:05.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9251 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc0ad7e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:05.715 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31427 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:05.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:07:05.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc047360 2021-03-09T10:07:05.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc047360 2021-03-09T10:07:05.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.708+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:05.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.708+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:05.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:05.708+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:10.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:10.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:10.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9251 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc0ad7e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:10.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9251 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc0ad7e0).stop 2021-03-09T10:07:10.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:10.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:10.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:10.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:10.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:10.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:10.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e040 con 0x7f21c4003cb0 2021-03-09T10:07:10.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc0138f0 2021-03-09T10:07:10.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:10.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:10.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:10.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:10.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:10.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:10.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:10.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:10.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:10.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ea00 con 0x7f21cc0138f0 2021-03-09T10:07:10.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc004c10 con 0x7f21cc0138f0 2021-03-09T10:07:10.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.704+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9262 cs=0 l=1 rev1=1 rx=0x7f21dc008c20 tx=0x7f21dc0ad770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:10.714 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:10.708+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31464 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:10.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0215f0 con 0x7f21cc0138f0 2021-03-09T10:07:10.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:07:10.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc0138f0 2021-03-09T10:07:10.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.712+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:10.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.712+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:10.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:10.712+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:15.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:15.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:15.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9262 cs=0 l=1 rev1=1 rx=0x7f21dc008c20 tx=0x7f21dc0ad770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:15.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9262 cs=0 l=1 rev1=1 rx=0x7f21dc008c20 tx=0x7f21dc0ad770).stop 2021-03-09T10:07:15.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:15.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:15.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:15.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:15.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:15.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:15.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0160a0 con 0x7f21c4003cb0 2021-03-09T10:07:15.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21cc047360 2021-03-09T10:07:15.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:15.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:15.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:15.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:15.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:15.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:15.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:15.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:15.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:15.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.708+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc086650 con 0x7f21cc047360 2021-03-09T10:07:15.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ea00 con 0x7f21cc047360 2021-03-09T10:07:15.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9272 cs=0 l=1 rev1=1 rx=0x7f21dc00d700 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:15.720 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:15.712+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31502 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:15.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.716+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00d220 con 0x7f21cc047360 2021-03-09T10:07:15.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.716+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0721a0 con 0x7f21cc047360 2021-03-09T10:07:15.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.716+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc047360 2021-03-09T10:07:15.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.716+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:15.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.720+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:15.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:15.720+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:20.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:20.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:20.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9272 cs=0 l=1 rev1=1 rx=0x7f21dc00d700 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:20.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9272 cs=0 l=1 rev1=1 rx=0x7f21dc00d700 tx=0x7f21dc077e50).stop 2021-03-09T10:07:20.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:20.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:20.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:20.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:20.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:20.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:20.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21c4003cb0 2021-03-09T10:07:20.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:07:20.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:20.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:20.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:20.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.712+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:20.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:20.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:20.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:20.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:20.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:20.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02e9d0 con 0x7f21cc0138f0 2021-03-09T10:07:20.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.716+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc086650 con 0x7f21cc0138f0 2021-03-09T10:07:20.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.720+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9285 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc033c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:20.727 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:20.720+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31539 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:20.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc033530 con 0x7f21cc0138f0 2021-03-09T10:07:20.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc033350 con 0x7f21cc0138f0 2021-03-09T10:07:20.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc0138f0 2021-03-09T10:07:20.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.728+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:20.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.728+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:20.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:20.728+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:24.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:24.427 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:25.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:25.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:25.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9285 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc033c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:25.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9285 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc033c10).stop 2021-03-09T10:07:25.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:25.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:25.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:25.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:25.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:25.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:25.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21c4003cb0 2021-03-09T10:07:25.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058010 con 0x7f21cc047360 2021-03-09T10:07:25.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:25.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:25.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:25.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:25.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:25.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:25.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:25.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:25.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:25.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:07:25.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e9d0 con 0x7f21cc047360 2021-03-09T10:07:25.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9295 cs=0 l=1 rev1=1 rx=0x7f21cc017800 tx=0x7f21dc0aba00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:25.732 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:25.724+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31577 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:25.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc033350 con 0x7f21cc047360 2021-03-09T10:07:25.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.732+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc033530 con 0x7f21cc047360 2021-03-09T10:07:25.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc019dd0 con 0x7f21cc047360 2021-03-09T10:07:25.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.736+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:25.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.736+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:25.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:25.736+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:30.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:30.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:30.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9295 cs=0 l=1 rev1=1 rx=0x7f21cc017800 tx=0x7f21dc0aba00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:30.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9295 cs=0 l=1 rev1=1 rx=0x7f21cc017800 tx=0x7f21dc0aba00).stop 2021-03-09T10:07:30.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:30.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:30.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:30.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:30.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:30.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:30.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0176f0 con 0x7f21c4003cb0 2021-03-09T10:07:30.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21cc0138f0 2021-03-09T10:07:30.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:30.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:30.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:30.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:30.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:30.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:30.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:30.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:30.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:30.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f50 con 0x7f21cc0138f0 2021-03-09T10:07:30.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e9d0 con 0x7f21cc0138f0 2021-03-09T10:07:30.738 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31614 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:30.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9309 cs=0 l=1 rev1=1 rx=0x7f21dc09bc40 tx=0x7f21dc02b2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:30.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.732+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083e60 con 0x7f21cc0138f0 2021-03-09T10:07:30.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc033530 con 0x7f21cc0138f0 2021-03-09T10:07:30.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.744+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc019da0 con 0x7f21cc0138f0 2021-03-09T10:07:30.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.744+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:30.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.744+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:30.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:30.744+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:35.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:35.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:35.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9309 cs=0 l=1 rev1=1 rx=0x7f21dc09bc40 tx=0x7f21dc02b2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9309 cs=0 l=1 rev1=1 rx=0x7f21dc09bc40 tx=0x7f21dc02b2a0).stop 2021-03-09T10:07:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:35.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:35.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:35.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056580 con 0x7f21cc005e30 2021-03-09T10:07:35.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012270 con 0x7f21c4003cb0 2021-03-09T10:07:35.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21cc047360 2021-03-09T10:07:35.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:35.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:35.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:35.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:35.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:35.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:35.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:35.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:35.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:35.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:07:35.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e9d0 con 0x7f21cc047360 2021-03-09T10:07:35.745 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:35.736+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31652 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:35.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.740+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9319 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc097f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:35.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083e60 con 0x7f21cc047360 2021-03-09T10:07:35.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.744+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083e60 con 0x7f21cc047360 2021-03-09T10:07:35.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.744+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc019da0 con 0x7f21cc047360 2021-03-09T10:07:35.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.744+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:35.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.744+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:35.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:35.744+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:40.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:40.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9319 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc097f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9319 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc097f10).stop 2021-03-09T10:07:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:40.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:40.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0176f0 con 0x7f21cc005e30 2021-03-09T10:07:40.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21c4003cb0 2021-03-09T10:07:40.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058010 con 0x7f21cc0138f0 2021-03-09T10:07:40.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:40.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:40.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:40.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:40.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.740+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f50 con 0x7f21cc0138f0 2021-03-09T10:07:40.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e9d0 con 0x7f21cc0138f0 2021-03-09T10:07:40.753 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31689 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:40.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.744+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9328 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc0ab720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:40.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b6ec0 con 0x7f21cc0138f0 2021-03-09T10:07:40.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6ce0 con 0x7f21cc0138f0 2021-03-09T10:07:40.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0196c0 con 0x7f21cc0138f0 2021-03-09T10:07:40.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.748+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:40.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.752+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:40.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:40.752+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:45.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:45.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:45.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9328 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc0ab720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:45.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9328 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc0ab720).stop 2021-03-09T10:07:45.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:45.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:45.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:45.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:45.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:45.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035930 con 0x7f21cc005e30 2021-03-09T10:07:45.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0176f0 con 0x7f21c4003cb0 2021-03-09T10:07:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058010 con 0x7f21cc047360 2021-03-09T10:07:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:07:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e9d0 con 0x7f21cc047360 2021-03-09T10:07:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.748+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9338 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc02b230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:45.757 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:45.752+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31727 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:45.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.752+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:07:45.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.760+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc047360 2021-03-09T10:07:45.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.760+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc061000 con 0x7f21cc047360 2021-03-09T10:07:45.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.760+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:45.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.760+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:45.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:45.760+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:50.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:50.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:50.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9338 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc02b230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:50.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9338 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc02b230).stop 2021-03-09T10:07:50.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:50.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:50.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:50.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:50.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:50.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0176f0 con 0x7f21cc005e30 2021-03-09T10:07:50.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:07:50.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc055f60 con 0x7f21cc0138f0 2021-03-09T10:07:50.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:50.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:50.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:50.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:50.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:50.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:50.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:50.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:50.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:50.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0846c0 con 0x7f21cc0138f0 2021-03-09T10:07:50.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02e9d0 con 0x7f21cc0138f0 2021-03-09T10:07:50.763 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31764 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:50.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.756+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9352 cs=0 l=1 rev1=1 rx=0x7f21dc0b34e0 tx=0x7f21dc06aba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:50.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.760+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:07:50.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.764+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:07:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.768+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0196c0 con 0x7f21cc0138f0 2021-03-09T10:07:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.768+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:50.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.768+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:50.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:50.768+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:07:54.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:54.483 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:07:55.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:55.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:55.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9352 cs=0 l=1 rev1=1 rx=0x7f21dc0b34e0 tx=0x7f21dc06aba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:55.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9352 cs=0 l=1 rev1=1 rx=0x7f21dc0b34e0 tx=0x7f21dc06aba0).stop 2021-03-09T10:07:55.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:55.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:55.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:55.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:55.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:55.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036340 con 0x7f21cc005e30 2021-03-09T10:07:55.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:07:55.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044f40 con 0x7f21cc047360 2021-03-09T10:07:55.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:07:55.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:55.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:07:55.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:55.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:55.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:55.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:55.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:55.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:55.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0450b0 con 0x7f21cc047360 2021-03-09T10:07:55.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084530 con 0x7f21cc047360 2021-03-09T10:07:55.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.759+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9358 cs=0 l=1 rev1=1 rx=0x7f21e4055570 tx=0x7f21e4055260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:55.771 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:07:55.763+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31802 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:07:55.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.763+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404eea0 con 0x7f21cc047360 2021-03-09T10:07:55.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.763+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016e30 con 0x7f21cc047360 2021-03-09T10:07:55.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc047360 2021-03-09T10:07:55.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.767+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:07:55.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.767+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:07:55.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:07:55.767+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:00.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:00.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:00.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9358 cs=0 l=1 rev1=1 rx=0x7f21e4055570 tx=0x7f21e4055260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:00.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9358 cs=0 l=1 rev1=1 rx=0x7f21e4055570 tx=0x7f21e4055260).stop 2021-03-09T10:08:00.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:00.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:00.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:00.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:00.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:00.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc00 con 0x7f21cc005e30 2021-03-09T10:08:00.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044f40 con 0x7f21c4003cb0 2021-03-09T10:08:00.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21cc0138f0 2021-03-09T10:08:00.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:00.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:00.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:00.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:00.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:00.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:00.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:00.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:00.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:00.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041050 con 0x7f21cc0138f0 2021-03-09T10:08:00.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0450b0 con 0x7f21cc0138f0 2021-03-09T10:08:00.779 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31839 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:00.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9373 cs=0 l=1 rev1=1 rx=0x7f21e409cfd0 tx=0x7f21e400d130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:00.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.771+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031230 con 0x7f21cc0138f0 2021-03-09T10:08:00.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.771+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404ec90 con 0x7f21cc0138f0 2021-03-09T10:08:00.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031230 con 0x7f21cc0138f0 2021-03-09T10:08:00.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.775+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:00.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.775+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:00.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:00.775+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:05.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:05.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:05.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9373 cs=0 l=1 rev1=1 rx=0x7f21e409cfd0 tx=0x7f21e400d130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:05.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9373 cs=0 l=1 rev1=1 rx=0x7f21e409cfd0 tx=0x7f21e400d130).stop 2021-03-09T10:08:05.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:05.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:05.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:05.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:05.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:05.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc005e30 2021-03-09T10:08:05.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cbb0 con 0x7f21c4003cb0 2021-03-09T10:08:05.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0450b0 con 0x7f21cc047360 2021-03-09T10:08:05.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.771+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:05.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:05.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:05.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:05.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:05.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:05.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:05.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:05.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:05.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0411e0 con 0x7f21cc047360 2021-03-09T10:08:05.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041050 con 0x7f21cc047360 2021-03-09T10:08:05.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.775+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9379 cs=0 l=1 rev1=1 rx=0x7f21e40a6040 tx=0x7f21e4016b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:05.783 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:05.779+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31877 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:05.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc047360 2021-03-09T10:08:05.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc047360 2021-03-09T10:08:05.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031230 con 0x7f21cc047360 2021-03-09T10:08:05.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.783+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:05.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.783+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:05.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:05.783+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:10.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:10.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:10.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9379 cs=0 l=1 rev1=1 rx=0x7f21e40a6040 tx=0x7f21e4016b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:10.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9379 cs=0 l=1 rev1=1 rx=0x7f21e40a6040 tx=0x7f21e4016b90).stop 2021-03-09T10:08:10.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:10.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:10.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:10.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:10.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:10.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d440 con 0x7f21cc005e30 2021-03-09T10:08:10.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0450b0 con 0x7f21c4003cb0 2021-03-09T10:08:10.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095130 con 0x7f21cc0138f0 2021-03-09T10:08:10.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:10.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:10.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:10.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:10.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:10.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:10.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:10.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:10.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:10.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0560d0 con 0x7f21cc0138f0 2021-03-09T10:08:10.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0411e0 con 0x7f21cc0138f0 2021-03-09T10:08:10.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9390 cs=0 l=1 rev1=1 rx=0x7f21cc040450 tx=0x7f21e402b5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:10.787 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:10.783+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31913 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:10.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.787+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007de0 con 0x7f21cc0138f0 2021-03-09T10:08:10.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.787+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc0138f0 2021-03-09T10:08:10.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.787+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40552d0 con 0x7f21cc0138f0 2021-03-09T10:08:10.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.787+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:10.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.791+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:10.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:10.791+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:15.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:15.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:15.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9390 cs=0 l=1 rev1=1 rx=0x7f21cc040450 tx=0x7f21e402b5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:15.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 secure :-1 s=READY pgs=9390 cs=0 l=1 rev1=1 rx=0x7f21cc040450 tx=0x7f21e402b5f0).stop 2021-03-09T10:08:15.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc055a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:15.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc055a70 unknown :-1 s=CLOSED pgs=9390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:15.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:15.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:15.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:15.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0450b0 con 0x7f21cc005e30 2021-03-09T10:08:15.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095130 con 0x7f21c4003cb0 2021-03-09T10:08:15.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d400 con 0x7f21cc047360 2021-03-09T10:08:15.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:15.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:15.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:15.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:15.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.783+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:15.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:15.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:15.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:15.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:15.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:08:15.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0846c0 con 0x7f21cc047360 2021-03-09T10:08:15.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.787+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7f21dc02b000 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:15.793 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:15.791+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31952 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:15.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.791+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc047360 2021-03-09T10:08:15.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.791+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c890 con 0x7f21cc047360 2021-03-09T10:08:15.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c510 con 0x7f21cc047360 2021-03-09T10:08:15.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.795+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:15.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.795+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:15.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:15.795+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:20.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:20.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:20.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7f21dc02b000 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:20.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7f21dc02b000 tx=0x7f21dc009810).stop 2021-03-09T10:08:20.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:20.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:20.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:20.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:20.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:20.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0567d0 con 0x7f21cc005e30 2021-03-09T10:08:20.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0450b0 con 0x7f21c4003cb0 2021-03-09T10:08:20.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045250 con 0x7f21cc0138f0 2021-03-09T10:08:20.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:20.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:20.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:20.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:20.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:20.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:20.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:20.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:20.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:20.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d400 con 0x7f21cc0138f0 2021-03-09T10:08:20.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.791+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0560d0 con 0x7f21cc0138f0 2021-03-09T10:08:20.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.795+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040e90 secure :-1 s=READY pgs=9411 cs=0 l=1 rev1=1 rx=0x7f21e4027e00 tx=0x7f21e408f250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:20.803 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:20.795+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31988 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:20.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.799+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402d230 con 0x7f21cc0138f0 2021-03-09T10:08:20.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.799+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e30 con 0x7f21cc0138f0 2021-03-09T10:08:20.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.799+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:08:20.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.799+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:20.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.803+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:20.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:20.803+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:24.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:24.529 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:25.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:25.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:25.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040e90 secure :-1 s=READY pgs=9411 cs=0 l=1 rev1=1 rx=0x7f21e4027e00 tx=0x7f21e408f250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:25.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040e90 secure :-1 s=READY pgs=9411 cs=0 l=1 rev1=1 rx=0x7f21e4027e00 tx=0x7f21e408f250).stop 2021-03-09T10:08:25.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:25.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040e90 unknown :-1 s=CLOSED pgs=9411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:25.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:25.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:25.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:25.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:25.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:25.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0567d0 con 0x7f21cc005e30 2021-03-09T10:08:25.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045080 con 0x7f21c4003cb0 2021-03-09T10:08:25.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:08:25.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:25.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.795+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:25.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:25.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:25.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:25.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:25.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:25.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094120 con 0x7f21cc047360 2021-03-09T10:08:25.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02d400 con 0x7f21cc047360 2021-03-09T10:08:25.805 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:25.803+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32027 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:25.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9424 cs=0 l=1 rev1=1 rx=0x7f21e402da80 tx=0x7f21e408a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:25.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b0070 con 0x7f21cc047360 2021-03-09T10:08:25.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e30 con 0x7f21cc047360 2021-03-09T10:08:25.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc047360 2021-03-09T10:08:25.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.807+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:25.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.807+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:25.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:25.807+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:30.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:30.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:30.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9424 cs=0 l=1 rev1=1 rx=0x7f21e402da80 tx=0x7f21e408a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:30.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9424 cs=0 l=1 rev1=1 rx=0x7f21e402da80 tx=0x7f21e408a310).stop 2021-03-09T10:08:30.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:30.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:30.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:30.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:30.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:30.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc005e30 2021-03-09T10:08:30.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057870 con 0x7f21c4003cb0 2021-03-09T10:08:30.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0579b0 con 0x7f21cc0138f0 2021-03-09T10:08:30.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:30.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:30.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:30.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:30.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:30.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:30.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:30.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:30.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:30.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d210 con 0x7f21cc0138f0 2021-03-09T10:08:30.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094120 con 0x7f21cc0138f0 2021-03-09T10:08:30.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.807+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040df0 secure :-1 s=READY pgs=9432 cs=0 l=1 rev1=1 rx=0x7f21e40b4220 tx=0x7f21e4027e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:30.811 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:30.807+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32063 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:30.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:08:30.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024160 con 0x7f21cc0138f0 2021-03-09T10:08:30.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.811+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:08:30.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.811+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:30.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.815+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:30.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:30.815+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:35.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:35.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:35.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040df0 secure :-1 s=READY pgs=9432 cs=0 l=1 rev1=1 rx=0x7f21e40b4220 tx=0x7f21e4027e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:35.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040df0 secure :-1 s=READY pgs=9432 cs=0 l=1 rev1=1 rx=0x7f21e40b4220 tx=0x7f21e4027e30).stop 2021-03-09T10:08:35.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:35.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040df0 unknown :-1 s=CLOSED pgs=9432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:35.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:35.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:35.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:35.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc005e30 2021-03-09T10:08:35.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057870 con 0x7f21c4003cb0 2021-03-09T10:08:35.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0579b0 con 0x7f21cc047360 2021-03-09T10:08:35.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:35.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.807+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:35.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:35.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:35.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:35.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:35.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:35.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:35.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:35.816 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32102 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:35.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093f30 con 0x7f21cc047360 2021-03-09T10:08:35.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.811+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0846c0 con 0x7f21cc047360 2021-03-09T10:08:35.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9448 cs=0 l=1 rev1=1 rx=0x7f21dc02f340 tx=0x7f21dc0b33f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:35.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bb7d0 con 0x7f21cc047360 2021-03-09T10:08:35.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc047360 2021-03-09T10:08:35.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c510 con 0x7f21cc047360 2021-03-09T10:08:35.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.819+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:35.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.823+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:35.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:35.823+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:40.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:40.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:40.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9448 cs=0 l=1 rev1=1 rx=0x7f21dc02f340 tx=0x7f21dc0b33f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:40.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9448 cs=0 l=1 rev1=1 rx=0x7f21dc02f340 tx=0x7f21dc0b33f0).stop 2021-03-09T10:08:40.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:40.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:40.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:40.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:40.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:40.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc005e30 2021-03-09T10:08:40.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:40.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0410f0 con 0x7f21c4003cb0 2021-03-09T10:08:40.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:40.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041230 con 0x7f21cc0138f0 2021-03-09T10:08:40.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:40.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:40.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:40.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:40.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:40.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:40.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:40.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0844f0 con 0x7f21cc0138f0 2021-03-09T10:08:40.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093f30 con 0x7f21cc0138f0 2021-03-09T10:08:40.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040c00 secure :-1 s=READY pgs=9455 cs=0 l=1 rev1=1 rx=0x7f21dc0564f0 tx=0x7f21dc0810e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:40.823 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:40.819+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32138 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:40.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc0138f0 2021-03-09T10:08:40.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c310 con 0x7f21cc0138f0 2021-03-09T10:08:40.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:08:40.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.823+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:40.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.823+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:40.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:40.823+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:45.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:45.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:45.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040c00 secure :-1 s=READY pgs=9455 cs=0 l=1 rev1=1 rx=0x7f21dc0564f0 tx=0x7f21dc0810e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:45.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040c00 secure :-1 s=READY pgs=9455 cs=0 l=1 rev1=1 rx=0x7f21dc0564f0 tx=0x7f21dc0810e0).stop 2021-03-09T10:08:45.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc040c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:45.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc040c00 unknown :-1 s=CLOSED pgs=9455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:45.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:45.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:45.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:45.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc005e30 2021-03-09T10:08:45.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013090 con 0x7f21c4003cb0 2021-03-09T10:08:45.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0131d0 con 0x7f21cc047360 2021-03-09T10:08:45.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:45.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:45.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:45.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:45.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.819+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:45.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:45.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:45.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:45.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:45.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057870 con 0x7f21cc047360 2021-03-09T10:08:45.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0844f0 con 0x7f21cc047360 2021-03-09T10:08:45.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9464 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc062480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:45.828 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:45.827+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32177 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:45.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.827+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc047360 2021-03-09T10:08:45.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.827+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc047360 2021-03-09T10:08:45.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc047360 2021-03-09T10:08:45.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.831+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:45.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.831+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:45.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:45.831+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:50.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:50.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:50.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9464 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc062480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:50.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9464 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc062480).stop 2021-03-09T10:08:50.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:50.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:50.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:50.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:50.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:50.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057640 con 0x7f21cc005e30 2021-03-09T10:08:50.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:08:50.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040f00 con 0x7f21cc0138f0 2021-03-09T10:08:50.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:50.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:50.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:50.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:50.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:50.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:50.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:50.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:50.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:50.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041040 con 0x7f21cc0138f0 2021-03-09T10:08:50.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.826+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02d210 con 0x7f21cc0138f0 2021-03-09T10:08:50.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.830+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9475 cs=0 l=1 rev1=1 rx=0x7f21e4050000 tx=0x7f21e4033c70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:50.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.830+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21cc0138f0 2021-03-09T10:08:50.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.830+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0070 con 0x7f21cc0138f0 2021-03-09T10:08:50.837 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:50.830+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32213 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:50.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.834+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409b000 con 0x7f21cc0138f0 2021-03-09T10:08:50.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.834+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:50.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.838+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:50.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:50.838+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:08:54.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:54.598 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:08:55.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:55.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:55.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9475 cs=0 l=1 rev1=1 rx=0x7f21e4050000 tx=0x7f21e4033c70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:55.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9475 cs=0 l=1 rev1=1 rx=0x7f21e4050000 tx=0x7f21e4033c70).stop 2021-03-09T10:08:55.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:55.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=CLOSED pgs=9475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:55.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:55.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:55.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:55.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040f00 con 0x7f21cc005e30 2021-03-09T10:08:55.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057640 con 0x7f21c4003cb0 2021-03-09T10:08:55.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc047360 2021-03-09T10:08:55.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:08:55.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:55.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:08:55.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.830+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:55.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:55.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:55.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:55.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:55.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:55.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0849e0 con 0x7f21cc047360 2021-03-09T10:08:55.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041040 con 0x7f21cc047360 2021-03-09T10:08:55.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.834+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9486 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40087b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:55.842 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:08:55.838+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32252 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:08:55.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.838+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4036d30 con 0x7f21cc047360 2021-03-09T10:08:55.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.838+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4036d30 con 0x7f21cc047360 2021-03-09T10:08:55.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40afc80 con 0x7f21cc047360 2021-03-09T10:08:55.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.842+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:08:55.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.846+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:08:55.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:08:55.846+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:00.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:00.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:00.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9486 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40087b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:00.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9486 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e40087b0).stop 2021-03-09T10:09:00.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:00.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:00.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:00.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:00.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:00.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030a50 con 0x7f21cc005e30 2021-03-09T10:09:00.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057640 con 0x7f21c4003cb0 2021-03-09T10:09:00.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc0138f0 2021-03-09T10:09:00.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:00.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:00.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:00.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:00.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.838+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:00.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:00.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:00.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:00.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:00.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cd80 con 0x7f21cc0138f0 2021-03-09T10:09:00.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0849e0 con 0x7f21cc0138f0 2021-03-09T10:09:00.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9497 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e408f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:00.848 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:00.842+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32288 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:00.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.846+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4038ee0 con 0x7f21cc0138f0 2021-03-09T10:09:00.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.846+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4038ee0 con 0x7f21cc0138f0 2021-03-09T10:09:00.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:09:00.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.850+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:00.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.850+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:00.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:00.850+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:05.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:05.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:05.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9497 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e408f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:05.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9497 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e408f000).stop 2021-03-09T10:09:05.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:05.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=CLOSED pgs=9497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:05.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:05.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:05.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:05.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc005e30 2021-03-09T10:09:05.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057640 con 0x7f21c4003cb0 2021-03-09T10:09:05.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039410 con 0x7f21cc047360 2021-03-09T10:09:05.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:05.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:05.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:05.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:05.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:05.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:05.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:05.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:05.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:05.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039550 con 0x7f21cc047360 2021-03-09T10:09:05.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057870 con 0x7f21cc047360 2021-03-09T10:09:05.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.846+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9511 cs=0 l=1 rev1=1 rx=0x7f21cc02cd00 tx=0x7f21dc0098c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:05.853 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:05.850+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32327 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:05.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0622c0 con 0x7f21cc047360 2021-03-09T10:09:05.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0622c0 con 0x7f21cc047360 2021-03-09T10:09:05.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.858+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc047360 2021-03-09T10:09:05.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.858+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:05.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.858+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:05.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:05.858+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:10.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:10.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:10.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9511 cs=0 l=1 rev1=1 rx=0x7f21cc02cd00 tx=0x7f21dc0098c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:10.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 secure :-1 s=READY pgs=9511 cs=0 l=1 rev1=1 rx=0x7f21cc02cd00 tx=0x7f21dc0098c0).stop 2021-03-09T10:09:10.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:10.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a60 unknown :-1 s=CLOSED pgs=9511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:10.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:10.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:10.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:10.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039410 con 0x7f21cc005e30 2021-03-09T10:09:10.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e60 con 0x7f21c4003cb0 2021-03-09T10:09:10.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:10.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21cc0138f0 2021-03-09T10:09:10.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:10.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:10.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:10.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.850+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:10.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:10.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:10.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:10.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:10.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0575f0 con 0x7f21cc0138f0 2021-03-09T10:09:10.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039550 con 0x7f21cc0138f0 2021-03-09T10:09:10.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9519 cs=0 l=1 rev1=1 rx=0x7f21dc028ff0 tx=0x7f21dc00c290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:10.861 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32363 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:10.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc070000 con 0x7f21cc0138f0 2021-03-09T10:09:10.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0212d0 con 0x7f21cc0138f0 2021-03-09T10:09:10.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.858+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc056120 con 0x7f21cc0138f0 2021-03-09T10:09:10.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.858+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:10.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.862+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:10.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:10.862+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:15.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:15.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:15.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9519 cs=0 l=1 rev1=1 rx=0x7f21dc028ff0 tx=0x7f21dc00c290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:15.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 secure :-1 s=READY pgs=9519 cs=0 l=1 rev1=1 rx=0x7f21dc028ff0 tx=0x7f21dc00c290).stop 2021-03-09T10:09:15.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0844f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:15.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0844f0 unknown :-1 s=CLOSED pgs=9519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:15.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:15.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:15.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:15.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21cc005e30 2021-03-09T10:09:15.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f20 con 0x7f21c4003cb0 2021-03-09T10:09:15.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035e50 con 0x7f21cc047360 2021-03-09T10:09:15.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:15.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:15.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:15.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:15.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.854+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:15.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:15.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:15.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:15.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:15.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057870 con 0x7f21cc047360 2021-03-09T10:09:15.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cd80 con 0x7f21cc047360 2021-03-09T10:09:15.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.858+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a30 secure :-1 s=READY pgs=9529 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4055230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:15.872 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:15.862+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32402 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:15.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027e30 con 0x7f21cc047360 2021-03-09T10:09:15.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027e30 con 0x7f21cc047360 2021-03-09T10:09:15.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc047360 2021-03-09T10:09:15.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.870+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:15.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.870+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:15.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:15.870+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:20.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:20.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:20.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a30 secure :-1 s=READY pgs=9529 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4055230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:20.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a30 secure :-1 s=READY pgs=9529 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4055230).stop 2021-03-09T10:09:20.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc026a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:20.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc026a30 unknown :-1 s=CLOSED pgs=9529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:20.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:20.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:20.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:20.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21cc005e30 2021-03-09T10:09:20.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0844f0 con 0x7f21c4003cb0 2021-03-09T10:09:20.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084630 con 0x7f21cc0138f0 2021-03-09T10:09:20.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:20.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:20.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:20.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:20.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:20.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:20.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:20.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:20.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:20.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084770 con 0x7f21cc0138f0 2021-03-09T10:09:20.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0575f0 con 0x7f21cc0138f0 2021-03-09T10:09:20.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.866+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cb90 secure :-1 s=READY pgs=9539 cs=0 l=1 rev1=1 rx=0x7f21dc089fd0 tx=0x7f21dc081000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:20.871 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:20.866+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32437 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:20.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:09:20.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028c70 con 0x7f21cc0138f0 2021-03-09T10:09:20.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06ddd0 con 0x7f21cc0138f0 2021-03-09T10:09:20.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.870+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:20.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.870+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:20.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:20.870+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:24.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:24.645 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:25.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:25.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:25.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cb90 secure :-1 s=READY pgs=9539 cs=0 l=1 rev1=1 rx=0x7f21dc089fd0 tx=0x7f21dc081000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:25.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cb90 secure :-1 s=READY pgs=9539 cs=0 l=1 rev1=1 rx=0x7f21dc089fd0 tx=0x7f21dc081000).stop 2021-03-09T10:09:25.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01cb90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:25.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cb90 unknown :-1 s=CLOSED pgs=9539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:25.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:25.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc084900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:25.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:25.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b5e0 con 0x7f21cc005e30 2021-03-09T10:09:25.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21c4003cb0 2021-03-09T10:09:25.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f20 con 0x7f21cc047360 2021-03-09T10:09:25.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:25.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:25.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:25.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:25.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc084900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:25.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:25.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:25.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:25.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:25.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084df0 con 0x7f21cc047360 2021-03-09T10:09:25.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057870 con 0x7f21cc047360 2021-03-09T10:09:25.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.870+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc084900 secure :-1 s=READY pgs=9552 cs=0 l=1 rev1=1 rx=0x7f21cc012c10 tx=0x7f21e4016ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:25.877 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:25.874+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32476 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:25.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4053290 con 0x7f21cc047360 2021-03-09T10:09:25.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc047360 2021-03-09T10:09:25.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007da0 con 0x7f21cc047360 2021-03-09T10:09:25.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.882+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:25.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.882+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:25.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:25.882+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:30.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc084900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:30.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc084900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:30.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc084900 secure :-1 s=READY pgs=9552 cs=0 l=1 rev1=1 rx=0x7f21cc012c10 tx=0x7f21e4016ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:30.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc084900 secure :-1 s=READY pgs=9552 cs=0 l=1 rev1=1 rx=0x7f21cc012c10 tx=0x7f21e4016ab0).stop 2021-03-09T10:09:30.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc084900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:30.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc084900 unknown :-1 s=CLOSED pgs=9552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:30.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:30.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:30.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:30.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a2b0 con 0x7f21cc005e30 2021-03-09T10:09:30.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21c4003cb0 2021-03-09T10:09:30.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040570 con 0x7f21cc0138f0 2021-03-09T10:09:30.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:30.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:30.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:30.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:30.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.874+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:30.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:30.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:30.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:30.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:30.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cee0 con 0x7f21cc0138f0 2021-03-09T10:09:30.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084770 con 0x7f21cc0138f0 2021-03-09T10:09:30.883 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32512 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:30.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c9f0 secure :-1 s=READY pgs=9565 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc081140).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:30.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081210 con 0x7f21cc0138f0 2021-03-09T10:09:30.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:09:30.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc0138f0 2021-03-09T10:09:30.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.886+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:30.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.886+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:30.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:30.886+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:35.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:35.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:35.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c9f0 secure :-1 s=READY pgs=9565 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc081140).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:35.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c9f0 secure :-1 s=READY pgs=9565 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc081140).stop 2021-03-09T10:09:35.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:35.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c9f0 unknown :-1 s=CLOSED pgs=9565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:35.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:35.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:35.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:35.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e6e0 con 0x7f21cc005e30 2021-03-09T10:09:35.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21c4003cb0 2021-03-09T10:09:35.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040570 con 0x7f21cc047360 2021-03-09T10:09:35.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:35.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:35.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:35.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.878+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:35.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:35.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:35.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:35.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:35.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:35.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cee0 con 0x7f21cc047360 2021-03-09T10:09:35.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084770 con 0x7f21cc047360 2021-03-09T10:09:35.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049f70 secure :-1 s=READY pgs=9574 cs=0 l=1 rev1=1 rx=0x7f21dc086010 tx=0x7f21dc062f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:35.889 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:35.886+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32551 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:35.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc047360 2021-03-09T10:09:35.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc047360 2021-03-09T10:09:35.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00c510 con 0x7f21cc047360 2021-03-09T10:09:35.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.894+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:35.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.894+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:35.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:35.894+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:40.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc049f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:40.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc049f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:40.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049f70 secure :-1 s=READY pgs=9574 cs=0 l=1 rev1=1 rx=0x7f21dc086010 tx=0x7f21dc062f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:40.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049f70 secure :-1 s=READY pgs=9574 cs=0 l=1 rev1=1 rx=0x7f21dc086010 tx=0x7f21dc062f30).stop 2021-03-09T10:09:40.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc049f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:40.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049f70 unknown :-1 s=CLOSED pgs=9574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:40.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:40.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:40.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:40.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc044d70 con 0x7f21cc005e30 2021-03-09T10:09:40.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032350 con 0x7f21c4003cb0 2021-03-09T10:09:40.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a460 con 0x7f21cc0138f0 2021-03-09T10:09:40.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:40.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:40.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:40.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:40.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:40.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:40.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:40.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:40.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:40.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cd40 con 0x7f21cc0138f0 2021-03-09T10:09:40.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.886+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084df0 con 0x7f21cc0138f0 2021-03-09T10:09:40.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.890+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c850 secure :-1 s=READY pgs=9582 cs=0 l=1 rev1=1 rx=0x7f21e402b710 tx=0x7f21e40911a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:40.895 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:40.890+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32587 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:40.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005520 con 0x7f21cc0138f0 2021-03-09T10:09:40.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21cc0138f0 2021-03-09T10:09:40.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40289f0 con 0x7f21cc0138f0 2021-03-09T10:09:40.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.894+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:40.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.894+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:40.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:40.894+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:45.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:45.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:45.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c850 secure :-1 s=READY pgs=9582 cs=0 l=1 rev1=1 rx=0x7f21e402b710 tx=0x7f21e40911a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:45.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c850 secure :-1 s=READY pgs=9582 cs=0 l=1 rev1=1 rx=0x7f21e402b710 tx=0x7f21e40911a0).stop 2021-03-09T10:09:45.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:45.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c850 unknown :-1 s=CLOSED pgs=9582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:45.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:45.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:45.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:45.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056420 con 0x7f21cc005e30 2021-03-09T10:09:45.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21c4003cb0 2021-03-09T10:09:45.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040570 con 0x7f21cc047360 2021-03-09T10:09:45.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:45.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:45.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:45.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:45.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.889+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:45.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:45.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:45.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:45.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:45.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc044d20 con 0x7f21cc047360 2021-03-09T10:09:45.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084770 con 0x7f21cc047360 2021-03-09T10:09:45.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.893+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049d80 secure :-1 s=READY pgs=9592 cs=0 l=1 rev1=1 rx=0x7f21dc020600 tx=0x7f21dc056ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:45.900 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:45.897+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32626 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:45.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc047360 2021-03-09T10:09:45.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc056ce0 con 0x7f21cc047360 2021-03-09T10:09:45.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc047360 2021-03-09T10:09:45.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.901+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:45.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.901+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:45.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:45.901+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:50.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:50.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:50.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049d80 secure :-1 s=READY pgs=9592 cs=0 l=1 rev1=1 rx=0x7f21dc020600 tx=0x7f21dc056ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:50.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049d80 secure :-1 s=READY pgs=9592 cs=0 l=1 rev1=1 rx=0x7f21dc020600 tx=0x7f21dc056ec0).stop 2021-03-09T10:09:50.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:50.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc049d80 unknown :-1 s=CLOSED pgs=9592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:50.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:50.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:50.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:50.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21cc005e30 2021-03-09T10:09:50.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056420 con 0x7f21c4003cb0 2021-03-09T10:09:50.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040570 con 0x7f21cc0138f0 2021-03-09T10:09:50.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:50.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:50.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:50.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:50.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:50.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:50.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:50.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:50.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc045bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:50.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084910 con 0x7f21cc0138f0 2021-03-09T10:09:50.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.897+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cd40 con 0x7f21cc0138f0 2021-03-09T10:09:50.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.901+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00be60 secure :-1 s=READY pgs=9605 cs=0 l=1 rev1=1 rx=0x7f21cc0677a0 tx=0x7f21e4016e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:50.907 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:50.901+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32662 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:50.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21cc0138f0 2021-03-09T10:09:50.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21cc0138f0 2021-03-09T10:09:50.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:09:50.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.909+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:50.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.909+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:50.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:50.909+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:51.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 msgr2=0x7f21cc039b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900002912) for more than 900000000 us, fault. 2021-03-09T10:09:51.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 0x7f21cc039b70 secure :-1 s=READY pgs=7307 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e00b0000).stop 2021-03-09T10:09:51.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21cc042f20 2021-03-09T10:09:51.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T10:09:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc042f20 0x7f21cc039b70 unknown :-1 s=CLOSED pgs=7307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:09:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 0x7f21cc049d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 0x7f21cc049d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:51.989+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 0x7f21cc049d80 secure :-1 s=READY pgs=9114 cs=0 l=1 rev1=1 rx=0x7f21cc030660 tx=0x7f21dc056ec0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:54.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:54.694 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:09:55.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00be60 secure :-1 s=READY pgs=9605 cs=0 l=1 rev1=1 rx=0x7f21cc0677a0 tx=0x7f21e4016e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00be60 secure :-1 s=READY pgs=9605 cs=0 l=1 rev1=1 rx=0x7f21cc0677a0 tx=0x7f21e4016e30).stop 2021-03-09T10:09:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00be60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00be60 unknown :-1 s=CLOSED pgs=9605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:55.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:55.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:55.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:55.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21cc005e30 2021-03-09T10:09:55.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040570 con 0x7f21c4003cb0 2021-03-09T10:09:55.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017790 con 0x7f21cc042f20 2021-03-09T10:09:55.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:09:55.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:55.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:09:55.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:55.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.905+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:55.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:55.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:55.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:55.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:55.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015ce0 con 0x7f21cc042f20 2021-03-09T10:09:55.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084910 con 0x7f21cc042f20 2021-03-09T10:09:55.914 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32701 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:09:55.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045bd0 secure :-1 s=READY pgs=9618 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e4024540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:55.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027eb0 con 0x7f21cc042f20 2021-03-09T10:09:55.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027eb0 con 0x7f21cc042f20 2021-03-09T10:09:55.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21cc042f20 2021-03-09T10:09:55.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.917+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:09:55.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.917+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:09:55.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:09:55.917+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:00.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045bd0 secure :-1 s=READY pgs=9618 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e4024540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045bd0 secure :-1 s=READY pgs=9618 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e4024540).stop 2021-03-09T10:10:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:00.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=9618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:00.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:00.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:00.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:00.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040570 con 0x7f21cc005e30 2021-03-09T10:10:00.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21c4003cb0 2021-03-09T10:10:00.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056400 con 0x7f21cc0138f0 2021-03-09T10:10:00.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:00.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015ce0 con 0x7f21cc0138f0 2021-03-09T10:10:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044d20 con 0x7f21cc0138f0 2021-03-09T10:10:00.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.913+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00bdc0 secure :-1 s=READY pgs=9624 cs=0 l=1 rev1=1 rx=0x7f21dc0726b0 tx=0x7f21dc062f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:00.920 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:00.917+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32737 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:00.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0abe70 con 0x7f21cc0138f0 2021-03-09T10:10:00.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc070000 con 0x7f21cc0138f0 2021-03-09T10:10:00.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:10:00.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.921+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:00.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.921+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:00.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:00.921+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:05.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:05.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:05.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00bdc0 secure :-1 s=READY pgs=9624 cs=0 l=1 rev1=1 rx=0x7f21dc0726b0 tx=0x7f21dc062f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:05.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00bdc0 secure :-1 s=READY pgs=9624 cs=0 l=1 rev1=1 rx=0x7f21dc0726b0 tx=0x7f21dc062f30).stop 2021-03-09T10:10:05.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00bdc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:05.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00bdc0 unknown :-1 s=CLOSED pgs=9624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:05.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:05.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:05.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:05.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057380 con 0x7f21cc005e30 2021-03-09T10:10:05.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056400 con 0x7f21c4003cb0 2021-03-09T10:10:05.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040540 con 0x7f21cc042f20 2021-03-09T10:10:05.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:05.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:05.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:05.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:05.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.917+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:05.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:05.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:05.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:05.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:05.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015ce0 con 0x7f21cc042f20 2021-03-09T10:10:05.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc042f20 2021-03-09T10:10:05.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 secure :-1 s=READY pgs=9636 cs=0 l=1 rev1=1 rx=0x7f21e000c010 tx=0x7f21e000b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:05.927 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:05.921+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32775 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:05.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:10:05.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:10:05.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.933+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:10:05.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.933+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:05.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.933+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:05.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:05.933+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:10.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:10.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:10.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 secure :-1 s=READY pgs=9636 cs=0 l=1 rev1=1 rx=0x7f21e000c010 tx=0x7f21e000b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:10.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 secure :-1 s=READY pgs=9636 cs=0 l=1 rev1=1 rx=0x7f21e000c010 tx=0x7f21e000b000).stop 2021-03-09T10:10:10.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:10.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 unknown :-1 s=CLOSED pgs=9636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:10.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:10.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0852e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:10.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:10.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045f50 con 0x7f21cc005e30 2021-03-09T10:10:10.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056400 con 0x7f21c4003cb0 2021-03-09T10:10:10.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032350 con 0x7f21cc0138f0 2021-03-09T10:10:10.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:10.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:10.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:10.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:10.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0852e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:10.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:10.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:10.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:10.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:10.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015ce0 con 0x7f21cc0138f0 2021-03-09T10:10:10.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.925+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084910 con 0x7f21cc0138f0 2021-03-09T10:10:10.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.929+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0852e0 secure :-1 s=READY pgs=9647 cs=0 l=1 rev1=1 rx=0x7f21e408a2c0 tx=0x7f21e4016b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:10.934 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:10.929+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32812 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:10.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.929+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4038ef0 con 0x7f21cc0138f0 2021-03-09T10:10:10.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4059000 con 0x7f21cc0138f0 2021-03-09T10:10:10.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21cc0138f0 2021-03-09T10:10:10.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.937+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:10.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.937+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:10.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:10.937+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:15.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0852e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:15.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0852e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:15.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0852e0 secure :-1 s=READY pgs=9647 cs=0 l=1 rev1=1 rx=0x7f21e408a2c0 tx=0x7f21e4016b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:15.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0852e0 secure :-1 s=READY pgs=9647 cs=0 l=1 rev1=1 rx=0x7f21e408a2c0 tx=0x7f21e4016b90).stop 2021-03-09T10:10:15.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0852e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:15.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0852e0 unknown :-1 s=CLOSED pgs=9647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:15.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:15.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:15.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:15.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc086570 con 0x7f21cc005e30 2021-03-09T10:10:15.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fa0 con 0x7f21c4003cb0 2021-03-09T10:10:15.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056400 con 0x7f21cc042f20 2021-03-09T10:10:15.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:15.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:15.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:15.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:15.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:15.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:15.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:15.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:15.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:15.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04da60 con 0x7f21cc042f20 2021-03-09T10:10:15.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc042f20 2021-03-09T10:10:15.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.933+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 secure :-1 s=READY pgs=9656 cs=0 l=1 rev1=1 rx=0x7f21e0013880 tx=0x7f21e00037b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:15.941 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:15.937+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32850 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:15.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:10:15.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:10:15.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:10:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.945+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.945+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:15.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:15.945+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:20.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:20.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:20.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 secure :-1 s=READY pgs=9656 cs=0 l=1 rev1=1 rx=0x7f21e0013880 tx=0x7f21e00037b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:20.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 secure :-1 s=READY pgs=9656 cs=0 l=1 rev1=1 rx=0x7f21e0013880 tx=0x7f21e00037b0).stop 2021-03-09T10:10:20.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:20.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084df0 unknown :-1 s=CLOSED pgs=9656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:20.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:20.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:20.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:20.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a410 con 0x7f21cc005e30 2021-03-09T10:10:20.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094600 con 0x7f21c4003cb0 2021-03-09T10:10:20.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:20.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fa0 con 0x7f21cc0138f0 2021-03-09T10:10:20.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:20.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:20.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:20.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:20.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:20.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.937+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:20.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.941+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:20.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.941+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:20.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.941+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0852e0 con 0x7f21cc0138f0 2021-03-09T10:10:20.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.941+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084910 con 0x7f21cc0138f0 2021-03-09T10:10:20.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.941+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c4c0 secure :-1 s=READY pgs=9667 cs=0 l=1 rev1=1 rx=0x7f21e40225c0 tx=0x7f21e401f2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:20.948 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:20.941+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32887 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:20.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc0138f0 2021-03-09T10:10:20.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc0138f0 2021-03-09T10:10:20.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4022280 con 0x7f21cc0138f0 2021-03-09T10:10:20.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.949+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:20.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.949+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:20.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:20.949+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:24.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:24.741 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:25.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:25.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:25.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c4c0 secure :-1 s=READY pgs=9667 cs=0 l=1 rev1=1 rx=0x7f21e40225c0 tx=0x7f21e401f2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:25.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c4c0 secure :-1 s=READY pgs=9667 cs=0 l=1 rev1=1 rx=0x7f21e40225c0 tx=0x7f21e401f2e0).stop 2021-03-09T10:10:25.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:25.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c4c0 unknown :-1 s=CLOSED pgs=9667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:25.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:25.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:25.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:25.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a410 con 0x7f21cc005e30 2021-03-09T10:10:25.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017840 con 0x7f21c4003cb0 2021-03-09T10:10:25.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:25.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fa0 con 0x7f21cc042f20 2021-03-09T10:10:25.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:25.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:25.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:25.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:25.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:25.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:25.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:25.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:25.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:10:25.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.945+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0852e0 con 0x7f21cc042f20 2021-03-09T10:10:25.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.949+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 secure :-1 s=READY pgs=9678 cs=0 l=1 rev1=1 rx=0x7f21cc012cb0 tx=0x7f21e4055840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:25.954 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:25.949+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32925 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:25.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T10:10:25.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401f930 con 0x7f21cc042f20 2021-03-09T10:10:25.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027ca0 con 0x7f21cc042f20 2021-03-09T10:10:25.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.957+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:25.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.957+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:25.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:25.957+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:30.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:30.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:30.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 secure :-1 s=READY pgs=9678 cs=0 l=1 rev1=1 rx=0x7f21cc012cb0 tx=0x7f21e4055840).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:30.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 secure :-1 s=READY pgs=9678 cs=0 l=1 rev1=1 rx=0x7f21cc012cb0 tx=0x7f21e4055840).stop 2021-03-09T10:10:30.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:30.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 unknown :-1 s=CLOSED pgs=9678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:30.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:30.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:30.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:30.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a410 con 0x7f21cc005e30 2021-03-09T10:10:30.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0179e0 con 0x7f21c4003cb0 2021-03-09T10:10:30.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007f70 con 0x7f21cc0138f0 2021-03-09T10:10:30.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:30.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:30.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:30.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:30.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:30.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:30.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:30.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:30.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:30.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0852e0 con 0x7f21cc0138f0 2021-03-09T10:10:30.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.953+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:10:30.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.957+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 secure :-1 s=READY pgs=9691 cs=0 l=1 rev1=1 rx=0x7f21e4016b60 tx=0x7f21e40489a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:30.962 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:30.957+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32962 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:30.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4048790 con 0x7f21cc0138f0 2021-03-09T10:10:30.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4048790 con 0x7f21cc0138f0 2021-03-09T10:10:30.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4022280 con 0x7f21cc0138f0 2021-03-09T10:10:30.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.965+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:30.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:30.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:30.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:35.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:35.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:35.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 secure :-1 s=READY pgs=9691 cs=0 l=1 rev1=1 rx=0x7f21e4016b60 tx=0x7f21e40489a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:35.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 secure :-1 s=READY pgs=9691 cs=0 l=1 rev1=1 rx=0x7f21e4016b60 tx=0x7f21e40489a0).stop 2021-03-09T10:10:35.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:35.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 unknown :-1 s=CLOSED pgs=9691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:35.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:35.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:35.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:35.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041290 con 0x7f21cc005e30 2021-03-09T10:10:35.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a410 con 0x7f21c4003cb0 2021-03-09T10:10:35.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:35.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d5e0 con 0x7f21cc042f20 2021-03-09T10:10:35.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:35.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:35.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:35.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:35.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:35.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:35.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:35.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:35.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d230 con 0x7f21cc042f20 2021-03-09T10:10:35.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0852e0 con 0x7f21cc042f20 2021-03-09T10:10:35.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.960+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 secure :-1 s=READY pgs=9699 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40b0070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:35.969 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:35.964+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33000 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:35.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5ec0 con 0x7f21cc042f20 2021-03-09T10:10:35.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5ec0 con 0x7f21cc042f20 2021-03-09T10:10:35.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:10:35.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.968+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:35.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:35.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:35.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:40.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:40.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:40.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 secure :-1 s=READY pgs=9699 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40b0070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:40.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 secure :-1 s=READY pgs=9699 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40b0070).stop 2021-03-09T10:10:40.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:40.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085470 unknown :-1 s=CLOSED pgs=9699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:40.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:40.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:40.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:40.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041290 con 0x7f21cc005e30 2021-03-09T10:10:40.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ea0 con 0x7f21c4003cb0 2021-03-09T10:10:40.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089500 con 0x7f21cc0138f0 2021-03-09T10:10:40.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:40.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:40.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:40.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:40.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:40.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:40.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:40.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:40.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:40.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084910 con 0x7f21cc0138f0 2021-03-09T10:10:40.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d230 con 0x7f21cc0138f0 2021-03-09T10:10:40.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.968+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=9710 cs=0 l=1 rev1=1 rx=0x7f21e402dc50 tx=0x7f21e40244c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:40.976 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:40.972+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33037 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:40.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402d230 con 0x7f21cc0138f0 2021-03-09T10:10:40.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404f320 con 0x7f21cc0138f0 2021-03-09T10:10:40.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:10:40.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.976+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:40.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.976+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:40.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:40.976+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:45.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:45.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:45.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=9710 cs=0 l=1 rev1=1 rx=0x7f21e402dc50 tx=0x7f21e40244c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:45.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=9710 cs=0 l=1 rev1=1 rx=0x7f21e402dc50 tx=0x7f21e40244c0).stop 2021-03-09T10:10:45.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:45.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=CLOSED pgs=9710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:45.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:45.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:45.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:45.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a410 con 0x7f21cc005e30 2021-03-09T10:10:45.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9b0 con 0x7f21c4003cb0 2021-03-09T10:10:45.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0276a0 con 0x7f21cc042f20 2021-03-09T10:10:45.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:45.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:45.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:45.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:45.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.972+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:45.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:45.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:45.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:45.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:45.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030f70 con 0x7f21cc042f20 2021-03-09T10:10:45.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc044d20 con 0x7f21cc042f20 2021-03-09T10:10:45.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 secure :-1 s=READY pgs=9723 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:45.983 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:45.976+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33075 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:45.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:10:45.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:10:45.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:10:45.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.984+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:45.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:45.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:45.984+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:50.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc057320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:50.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc057320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:50.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 secure :-1 s=READY pgs=9723 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:50.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 secure :-1 s=READY pgs=9723 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc077de0).stop 2021-03-09T10:10:50.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc057320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:50.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 unknown :-1 s=CLOSED pgs=9723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:50.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:50.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:50.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:50.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0276a0 con 0x7f21cc005e30 2021-03-09T10:10:50.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b960 con 0x7f21c4003cb0 2021-03-09T10:10:50.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012030 con 0x7f21cc0138f0 2021-03-09T10:10:50.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:50.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:50.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:50.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:50.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:50.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:50.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:50.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:50.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:50.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089500 con 0x7f21cc0138f0 2021-03-09T10:10:50.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.980+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084910 con 0x7f21cc0138f0 2021-03-09T10:10:50.990 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:50.984+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33112 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:50.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=9736 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e409bda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:50.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40559d0 con 0x7f21cc0138f0 2021-03-09T10:10:50.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4005480 con 0x7f21cc0138f0 2021-03-09T10:10:50.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:10:50.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.992+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:50.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.992+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:50.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:50.992+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:10:54.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:54.781 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:10:55.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:55.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:55.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=9736 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e409bda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:55.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=9736 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e409bda0).stop 2021-03-09T10:10:55.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:55.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=CLOSED pgs=9736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:55.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:55.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:55.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:55.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:10:55.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d0b0 con 0x7f21c4003cb0 2021-03-09T10:10:55.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012000 con 0x7f21cc042f20 2021-03-09T10:10:55.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:10:55.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:55.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:10:55.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:55.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.984+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:55.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:55.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:55.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:55.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:55.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012170 con 0x7f21cc042f20 2021-03-09T10:10:55.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030f70 con 0x7f21cc042f20 2021-03-09T10:10:55.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 secure :-1 s=READY pgs=9742 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:55.996 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33150 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:10:55.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05ec00 con 0x7f21cc042f20 2021-03-09T10:10:55.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.988+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc042f20 2021-03-09T10:10:56.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:10:56.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.996+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:10:56.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.996+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:10:56.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:10:55.996+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:00.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc057320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:11:00.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc057320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:00.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 secure :-1 s=READY pgs=9742 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:00.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 secure :-1 s=READY pgs=9742 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009810).stop 2021-03-09T10:11:00.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc057320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:00.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc057320 unknown :-1 s=CLOSED pgs=9742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:00.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:00.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:00.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:00.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0124a0 con 0x7f21cc005e30 2021-03-09T10:11:00.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01bf90 con 0x7f21c4003cb0 2021-03-09T10:11:00.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c0d0 con 0x7f21cc0138f0 2021-03-09T10:11:00.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:00.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:00.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:00.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:00.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:01.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:01.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:01.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:01.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:01.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c210 con 0x7f21cc0138f0 2021-03-09T10:11:01.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.992+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04da60 con 0x7f21cc0138f0 2021-03-09T10:11:01.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.996+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096630 secure :-1 s=READY pgs=9755 cs=0 l=1 rev1=1 rx=0x7f21e000ffd0 tx=0x7f21e002a170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:01.002 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:00.996+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33187 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:01.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:00.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:11:01.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:01.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:11:01.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:01.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:11:01.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:01.004+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:01.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:01.004+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:01.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:01.004+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:06.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc096630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:06.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc096630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:06.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096630 secure :-1 s=READY pgs=9755 cs=0 l=1 rev1=1 rx=0x7f21e000ffd0 tx=0x7f21e002a170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:06.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096630 secure :-1 s=READY pgs=9755 cs=0 l=1 rev1=1 rx=0x7f21e000ffd0 tx=0x7f21e002a170).stop 2021-03-09T10:11:06.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc096630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:06.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096630 unknown :-1 s=CLOSED pgs=9755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:06.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:06.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:06.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:06.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c890 con 0x7f21cc005e30 2021-03-09T10:11:06.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0124a0 con 0x7f21c4003cb0 2021-03-09T10:11:06.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:11:06.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:06.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:06.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:06.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:06.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:05.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:06.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:06.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:06.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:06.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:06.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d0e0 con 0x7f21cc042f20 2021-03-09T10:11:06.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012170 con 0x7f21cc042f20 2021-03-09T10:11:06.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.000+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9766 cs=0 l=1 rev1=1 rx=0x7f21dc052580 tx=0x7f21dc003fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:06.009 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:06.004+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33225 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:06.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0059d0 con 0x7f21cc042f20 2021-03-09T10:11:06.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0057f0 con 0x7f21cc042f20 2021-03-09T10:11:06.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:11:06.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.012+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:06.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.012+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:06.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:06.012+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:11.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:11.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:11.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9766 cs=0 l=1 rev1=1 rx=0x7f21dc052580 tx=0x7f21dc003fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:11.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9766 cs=0 l=1 rev1=1 rx=0x7f21dc052580 tx=0x7f21dc003fb0).stop 2021-03-09T10:11:11.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:11.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=CLOSED pgs=9766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:11.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:11.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:11.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:11.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c890 con 0x7f21cc005e30 2021-03-09T10:11:11.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0281f0 con 0x7f21c4003cb0 2021-03-09T10:11:11.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:11:11.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:11.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:11.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:11.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:11.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:11.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:11.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:11.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:11.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:11.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc035ff0 con 0x7f21cc0138f0 2021-03-09T10:11:11.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d0e0 con 0x7f21cc0138f0 2021-03-09T10:11:11.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d270 secure :-1 s=READY pgs=9778 cs=0 l=1 rev1=1 rx=0x7f21cc0238b0 tx=0x7f21dc0057f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:11.015 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:11.012+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33262 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:11.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.012+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc050020 con 0x7f21cc0138f0 2021-03-09T10:11:11.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00af60 con 0x7f21cc0138f0 2021-03-09T10:11:11.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:11:11.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.020+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:11.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.020+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:11.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:11.020+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:16.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:16.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:16.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d270 secure :-1 s=READY pgs=9778 cs=0 l=1 rev1=1 rx=0x7f21cc0238b0 tx=0x7f21dc0057f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:16.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d270 secure :-1 s=READY pgs=9778 cs=0 l=1 rev1=1 rx=0x7f21cc0238b0 tx=0x7f21dc0057f0).stop 2021-03-09T10:11:16.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:16.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d270 unknown :-1 s=CLOSED pgs=9778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:16.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:16.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:16.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:16.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c890 con 0x7f21cc005e30 2021-03-09T10:11:16.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:11:16.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d0b0 con 0x7f21cc042f20 2021-03-09T10:11:16.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:16.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:16.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:16.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:16.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:16.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:16.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:16.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:16.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:16.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096440 con 0x7f21cc042f20 2021-03-09T10:11:16.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc035ff0 con 0x7f21cc042f20 2021-03-09T10:11:16.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9786 cs=0 l=1 rev1=1 rx=0x7f21dc066040 tx=0x7f21dc005cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:16.024 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:16.016+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:16.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0079b0 con 0x7f21cc042f20 2021-03-09T10:11:16.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0077d0 con 0x7f21cc042f20 2021-03-09T10:11:16.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc042f20 2021-03-09T10:11:16.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.024+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:16.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.024+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:16.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:16.024+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:21.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:21.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.016+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:21.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9786 cs=0 l=1 rev1=1 rx=0x7f21dc066040 tx=0x7f21dc005cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:21.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.016+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9786 cs=0 l=1 rev1=1 rx=0x7f21dc066040 tx=0x7f21dc005cd0).stop 2021-03-09T10:11:21.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:21.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=CLOSED pgs=9786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:21.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:21.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:21.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:21.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c890 con 0x7f21cc005e30 2021-03-09T10:11:21.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096ac0 con 0x7f21c4003cb0 2021-03-09T10:11:21.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:11:21.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:21.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:21.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:21.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:21.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:21.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:21.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:21.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:21.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:21.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d0e0 con 0x7f21cc0138f0 2021-03-09T10:11:21.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089500 con 0x7f21cc0138f0 2021-03-09T10:11:21.036 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33337 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:21.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0965d0 secure :-1 s=READY pgs=9800 cs=0 l=1 rev1=1 rx=0x7f21e401f080 tx=0x7f21e402b270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:21.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402b610 con 0x7f21cc0138f0 2021-03-09T10:11:21.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.024+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402b610 con 0x7f21cc0138f0 2021-03-09T10:11:21.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc0138f0 2021-03-09T10:11:21.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.032+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:21.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:21.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:21.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:24.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:24.849 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:26.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:26.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:26.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0965d0 secure :-1 s=READY pgs=9800 cs=0 l=1 rev1=1 rx=0x7f21e401f080 tx=0x7f21e402b270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:26.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0965d0 secure :-1 s=READY pgs=9800 cs=0 l=1 rev1=1 rx=0x7f21e401f080 tx=0x7f21e402b270).stop 2021-03-09T10:11:26.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0965d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:26.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0965d0 unknown :-1 s=CLOSED pgs=9800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:26.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:26.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:26.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:26.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c890 con 0x7f21cc005e30 2021-03-09T10:11:26.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01da30 con 0x7f21c4003cb0 2021-03-09T10:11:26.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04bc20 con 0x7f21cc042f20 2021-03-09T10:11:26.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:26.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:26.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:26.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:26.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:26.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:26.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:26.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:26.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:26.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d270 con 0x7f21cc042f20 2021-03-09T10:11:26.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c210 con 0x7f21cc042f20 2021-03-09T10:11:26.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.028+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9806 cs=0 l=1 rev1=1 rx=0x7f21e0077e40 tx=0x7f21e002a100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:26.036 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:26.032+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33375 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:26.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc042f20 2021-03-09T10:11:26.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc042f20 2021-03-09T10:11:26.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.036+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:11:26.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.036+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:26.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.036+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:26.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:26.036+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:31.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:31.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:31.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9806 cs=0 l=1 rev1=1 rx=0x7f21e0077e40 tx=0x7f21e002a100).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:31.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 secure :-1 s=READY pgs=9806 cs=0 l=1 rev1=1 rx=0x7f21e0077e40 tx=0x7f21e002a100).stop 2021-03-09T10:11:31.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01c3a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:31.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01c3a0 unknown :-1 s=CLOSED pgs=9806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:31.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:31.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:31.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:31.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023710 con 0x7f21cc005e30 2021-03-09T10:11:31.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:11:31.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046070 con 0x7f21cc0138f0 2021-03-09T10:11:31.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:31.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:31.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:31.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:31.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:31.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:31.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:31.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:31.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:31.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04bc20 con 0x7f21cc0138f0 2021-03-09T10:11:31.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d0e0 con 0x7f21cc0138f0 2021-03-09T10:11:31.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.035+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d400 secure :-1 s=READY pgs=9816 cs=0 l=1 rev1=1 rx=0x7f21e40b04d0 tx=0x7f21e40b0440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:31.043 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:31.035+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33412 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:31.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc0138f0 2021-03-09T10:11:31.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:11:31.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:11:31.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.043+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:31.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.043+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:31.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:31.043+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:36.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:11:36.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:36.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d400 secure :-1 s=READY pgs=9816 cs=0 l=1 rev1=1 rx=0x7f21e40b04d0 tx=0x7f21e40b0440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:36.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d400 secure :-1 s=READY pgs=9816 cs=0 l=1 rev1=1 rx=0x7f21e40b04d0 tx=0x7f21e40b0440).stop 2021-03-09T10:11:36.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:36.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d400 unknown :-1 s=CLOSED pgs=9816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:36.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:36.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:36.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:36.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:11:36.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0267d0 con 0x7f21c4003cb0 2021-03-09T10:11:36.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096ac0 con 0x7f21cc042f20 2021-03-09T10:11:36.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:36.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:36.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:36.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:36.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:36.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:36.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:36.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:36.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:36.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c190 con 0x7f21cc042f20 2021-03-09T10:11:36.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d270 con 0x7f21cc042f20 2021-03-09T10:11:36.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0965d0 secure :-1 s=READY pgs=9827 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:36.054 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:36.047+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33450 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:36.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21cc042f20 2021-03-09T10:11:36.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:11:36.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:11:36.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.051+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:36.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.051+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:36.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:36.051+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:41.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:41.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:41.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0965d0 secure :-1 s=READY pgs=9827 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:41.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0965d0 secure :-1 s=READY pgs=9827 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e007c5a0).stop 2021-03-09T10:11:41.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0965d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:41.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0965d0 unknown :-1 s=CLOSED pgs=9827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:41.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:41.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:41.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:41.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:11:41.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c680 con 0x7f21c4003cb0 2021-03-09T10:11:41.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c7c0 con 0x7f21cc0138f0 2021-03-09T10:11:41.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:41.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:41.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:41.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:41.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:41.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:41.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:41.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:41.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:41.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026630 con 0x7f21cc0138f0 2021-03-09T10:11:41.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096440 con 0x7f21cc0138f0 2021-03-09T10:11:41.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 secure :-1 s=READY pgs=9840 cs=0 l=1 rev1=1 rx=0x7f21cc057ad0 tx=0x7f21dc005b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:41.057 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:41.055+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33487 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:41.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc0138f0 2021-03-09T10:11:41.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc0138f0 2021-03-09T10:11:41.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc005860 con 0x7f21cc0138f0 2021-03-09T10:11:41.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.059+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:41.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:41.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:41.063+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:46.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:46.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:46.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 secure :-1 s=READY pgs=9840 cs=0 l=1 rev1=1 rx=0x7f21cc057ad0 tx=0x7f21dc005b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:46.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 secure :-1 s=READY pgs=9840 cs=0 l=1 rev1=1 rx=0x7f21cc057ad0 tx=0x7f21dc005b70).stop 2021-03-09T10:11:46.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:46.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 unknown :-1 s=CLOSED pgs=9840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:46.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:46.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:46.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:46.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030880 con 0x7f21cc005e30 2021-03-09T10:11:46.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c820 con 0x7f21c4003cb0 2021-03-09T10:11:46.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:11:46.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:46.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:46.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:46.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:46.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.055+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:46.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:46.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:46.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:46.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:46.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096930 con 0x7f21cc042f20 2021-03-09T10:11:46.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04bc20 con 0x7f21cc042f20 2021-03-09T10:11:46.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096440 secure :-1 s=READY pgs=9854 cs=0 l=1 rev1=1 rx=0x7f21e4053fd0 tx=0x7f21e4005da0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:46.063 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:46.059+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33525 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:46.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401f1f0 con 0x7f21cc042f20 2021-03-09T10:11:46.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024410 con 0x7f21cc042f20 2021-03-09T10:11:46.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:11:46.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.071+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:46.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.071+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:46.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:46.071+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:51.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:51.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:51.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096440 secure :-1 s=READY pgs=9854 cs=0 l=1 rev1=1 rx=0x7f21e4053fd0 tx=0x7f21e4005da0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:51.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096440 secure :-1 s=READY pgs=9854 cs=0 l=1 rev1=1 rx=0x7f21e4053fd0 tx=0x7f21e4005da0).stop 2021-03-09T10:11:51.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:51.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096440 unknown :-1 s=CLOSED pgs=9854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:51.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:51.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:51.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:51.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc009ca0 con 0x7f21cc005e30 2021-03-09T10:11:51.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030880 con 0x7f21c4003cb0 2021-03-09T10:11:51.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:11:51.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:51.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:51.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:51.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:51.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:51.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:51.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:51.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:51.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:51.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c680 con 0x7f21cc0138f0 2021-03-09T10:11:51.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026630 con 0x7f21cc0138f0 2021-03-09T10:11:51.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 secure :-1 s=READY pgs=9863 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc00ad10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:51.070 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33562 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:51.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00af50 con 0x7f21cc0138f0 2021-03-09T10:11:51.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00af50 con 0x7f21cc0138f0 2021-03-09T10:11:51.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc005d40 con 0x7f21cc0138f0 2021-03-09T10:11:51.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.075+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:51.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.075+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:51.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:51.075+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:11:54.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:54.892 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:11:56.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:56.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:56.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 secure :-1 s=READY pgs=9863 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc00ad10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:56.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 secure :-1 s=READY pgs=9863 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc00ad10).stop 2021-03-09T10:11:56.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:56.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c190 unknown :-1 s=CLOSED pgs=9863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:56.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:56.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:56.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:56.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:11:56.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008160 con 0x7f21cc005e30 2021-03-09T10:11:56.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:56.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02eab0 con 0x7f21c4003cb0 2021-03-09T10:11:56.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.067+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030a20 con 0x7f21cc042f20 2021-03-09T10:11:56.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:11:56.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:56.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:56.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:56.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:56.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:56.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:56.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0267d0 con 0x7f21cc042f20 2021-03-09T10:11:56.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.071+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c680 con 0x7f21cc042f20 2021-03-09T10:11:56.077 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:11:56.075+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33600 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:11:56.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.075+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9874 cs=0 l=1 rev1=1 rx=0x7f21cc024410 tx=0x7f21dc089c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:56.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:11:56.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:11:56.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21cc042f20 2021-03-09T10:11:56.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.079+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:11:56.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.083+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:11:56.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:11:56.083+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:01.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:01.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:01.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9874 cs=0 l=1 rev1=1 rx=0x7f21cc024410 tx=0x7f21dc089c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:01.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9874 cs=0 l=1 rev1=1 rx=0x7f21cc024410 tx=0x7f21dc089c20).stop 2021-03-09T10:12:01.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:01.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=CLOSED pgs=9874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:01.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:01.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:01.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:01.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008160 con 0x7f21cc005e30 2021-03-09T10:12:01.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02eab0 con 0x7f21c4003cb0 2021-03-09T10:12:01.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:12:01.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:01.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:01.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:01.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:01.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:01.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:01.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:01.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:01.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:01.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c190 con 0x7f21cc0138f0 2021-03-09T10:12:01.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.075+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084020 con 0x7f21cc0138f0 2021-03-09T10:12:01.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.079+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9880 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e007d640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:01.083 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:01.083+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33636 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:01.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0138f0 2021-03-09T10:12:01.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.087+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d1d0 con 0x7f21cc0138f0 2021-03-09T10:12:01.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.087+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:12:01.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.087+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:01.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.087+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:01.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:01.087+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:06.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:06.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:06.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9880 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e007d640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:06.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9880 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e007d640).stop 2021-03-09T10:12:06.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:06.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=9880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:06.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:06.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:06.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:06.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008160 con 0x7f21cc005e30 2021-03-09T10:12:06.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02eab0 con 0x7f21c4003cb0 2021-03-09T10:12:06.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf50 con 0x7f21cc042f20 2021-03-09T10:12:06.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:06.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:06.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:06.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:06.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.083+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:06.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc02fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:06.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc02fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:06.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:06.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:06.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04e0e0 con 0x7f21cc042f20 2021-03-09T10:12:06.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c190 con 0x7f21cc042f20 2021-03-09T10:12:06.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9896 cs=0 l=1 rev1=1 rx=0x7f21e009a830 tx=0x7f21e00a9ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:06.091 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:06.087+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33675 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:06.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:12:06.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d1d0 con 0x7f21cc042f20 2021-03-09T10:12:06.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a0000 con 0x7f21cc042f20 2021-03-09T10:12:06.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.095+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:06.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.095+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:06.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:06.095+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:11.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:11.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:11.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9896 cs=0 l=1 rev1=1 rx=0x7f21e009a830 tx=0x7f21e00a9ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:11.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9896 cs=0 l=1 rev1=1 rx=0x7f21e009a830 tx=0x7f21e00a9ef0).stop 2021-03-09T10:12:11.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:11.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=CLOSED pgs=9896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:11.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:11.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:11.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:11.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b770 con 0x7f21cc005e30 2021-03-09T10:12:11.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008160 con 0x7f21c4003cb0 2021-03-09T10:12:11.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf50 con 0x7f21cc0138f0 2021-03-09T10:12:11.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:11.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:11.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:11.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc08d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:11.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:11.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:11.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc08d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:11.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:11.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:11.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ea80 con 0x7f21cc0138f0 2021-03-09T10:12:11.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.091+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0267d0 con 0x7f21cc0138f0 2021-03-09T10:12:11.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.095+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 secure :-1 s=READY pgs=9905 cs=0 l=1 rev1=1 rx=0x7f21dc089b40 tx=0x7f21dc0800b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:11.099 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:11.095+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33711 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:11.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.095+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc0138f0 2021-03-09T10:12:11.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc0138f0 2021-03-09T10:12:11.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.099+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:12:11.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.099+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:11.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.103+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:11.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:11.103+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:16.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:16.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:16.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 secure :-1 s=READY pgs=9905 cs=0 l=1 rev1=1 rx=0x7f21dc089b40 tx=0x7f21dc0800b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:16.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 secure :-1 s=READY pgs=9905 cs=0 l=1 rev1=1 rx=0x7f21dc089b40 tx=0x7f21dc0800b0).stop 2021-03-09T10:12:16.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02fab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:16.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 unknown :-1 s=CLOSED pgs=9905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:16.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:16.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:16.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:16.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf50 con 0x7f21cc005e30 2021-03-09T10:12:16.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:12:16.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0267d0 con 0x7f21cc042f20 2021-03-09T10:12:16.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:16.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.094+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:16.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:16.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:16.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:16.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:16.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:16.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:16.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:16.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030ec0 con 0x7f21cc042f20 2021-03-09T10:12:16.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096930 con 0x7f21cc042f20 2021-03-09T10:12:16.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.098+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9913 cs=0 l=1 rev1=1 rx=0x7f21e407f980 tx=0x7f21e4091200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:16.106 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:16.102+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33750 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:16.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T10:12:16.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0240 con 0x7f21cc042f20 2021-03-09T10:12:16.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.106+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007d80 con 0x7f21cc042f20 2021-03-09T10:12:16.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.106+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:16.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.106+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:16.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:16.106+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:21.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:21.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:21.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9913 cs=0 l=1 rev1=1 rx=0x7f21e407f980 tx=0x7f21e4091200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:21.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9913 cs=0 l=1 rev1=1 rx=0x7f21e407f980 tx=0x7f21e4091200).stop 2021-03-09T10:12:21.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:21.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=CLOSED pgs=9913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:21.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:21.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:21.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:21.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:12:21.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf50 con 0x7f21c4003cb0 2021-03-09T10:12:21.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c820 con 0x7f21cc0138f0 2021-03-09T10:12:21.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:21.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:21.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:21.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:21.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:21.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:21.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:21.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:21.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:21.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026630 con 0x7f21cc0138f0 2021-03-09T10:12:21.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.102+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030ec0 con 0x7f21cc0138f0 2021-03-09T10:12:21.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.106+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7f21cc01e800 tx=0x7f21e4017bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:21.111 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:21.106+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33786 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:21.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc0138f0 2021-03-09T10:12:21.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0240 con 0x7f21cc0138f0 2021-03-09T10:12:21.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:12:21.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.114+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:21.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.114+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:21.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:21.114+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:24.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:24.940 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:26.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:26.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:26.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7f21cc01e800 tx=0x7f21e4017bd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:26.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7f21cc01e800 tx=0x7f21e4017bd0).stop 2021-03-09T10:12:26.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02fab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:26.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02fab0 unknown :-1 s=CLOSED pgs=9923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:26.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:26.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:26.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:26.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:12:26.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf50 con 0x7f21c4003cb0 2021-03-09T10:12:26.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085090 con 0x7f21cc042f20 2021-03-09T10:12:26.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:26.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:26.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:26.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:26.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:26.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:26.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:26.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:26.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:26.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091ab0 con 0x7f21cc042f20 2021-03-09T10:12:26.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026630 con 0x7f21cc042f20 2021-03-09T10:12:26.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.110+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9934 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e40af100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:26.118 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:26.114+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33825 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:26.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T10:12:26.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T10:12:26.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc042f20 2021-03-09T10:12:26.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.118+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:26.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.122+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:26.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:26.122+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:31.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:31.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:31.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9934 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e40af100).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:31.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 secure :-1 s=READY pgs=9934 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e40af100).stop 2021-03-09T10:12:31.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc08d180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:31.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc08d180 unknown :-1 s=CLOSED pgs=9934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:31.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:31.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:31.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:31.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:12:31.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041f80 con 0x7f21c4003cb0 2021-03-09T10:12:31.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf50 con 0x7f21cc0138f0 2021-03-09T10:12:31.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:31.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:31.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:31.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:31.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:31.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:31.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:31.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:31.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:31.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030d00 con 0x7f21cc0138f0 2021-03-09T10:12:31.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.118+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091ab0 con 0x7f21cc0138f0 2021-03-09T10:12:31.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.122+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9945 cs=0 l=1 rev1=1 rx=0x7f21e407f790 tx=0x7f21e408aaa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:31.126 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:31.122+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33861 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:31.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:12:31.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40af240 con 0x7f21cc0138f0 2021-03-09T10:12:31.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:12:31.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.130+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:31.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.134+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:31.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:31.134+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:36.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:36.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:36.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9945 cs=0 l=1 rev1=1 rx=0x7f21e407f790 tx=0x7f21e408aaa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:36.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9945 cs=0 l=1 rev1=1 rx=0x7f21e407f790 tx=0x7f21e408aaa0).stop 2021-03-09T10:12:36.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:36.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=9945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:36.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:36.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:36.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:36.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:12:36.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042120 con 0x7f21c4003cb0 2021-03-09T10:12:36.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085090 con 0x7f21cc042f20 2021-03-09T10:12:36.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:36.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:36.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:36.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.122+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:36.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:36.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:36.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:36.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:36.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:36.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc045bd0 con 0x7f21cc042f20 2021-03-09T10:12:36.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030d00 con 0x7f21cc042f20 2021-03-09T10:12:36.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.126+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 secure :-1 s=READY pgs=9957 cs=0 l=1 rev1=1 rx=0x7f21cc008030 tx=0x7f21e4017b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:36.134 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:36.130+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33900 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:36.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017e90 con 0x7f21cc042f20 2021-03-09T10:12:36.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017e90 con 0x7f21cc042f20 2021-03-09T10:12:36.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40af170 con 0x7f21cc042f20 2021-03-09T10:12:36.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.134+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:36.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.134+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:36.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:36.134+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:41.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:41.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:41.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 secure :-1 s=READY pgs=9957 cs=0 l=1 rev1=1 rx=0x7f21cc008030 tx=0x7f21e4017b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:41.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 secure :-1 s=READY pgs=9957 cs=0 l=1 rev1=1 rx=0x7f21cc008030 tx=0x7f21e4017b20).stop 2021-03-09T10:12:41.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02fab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:41.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 unknown :-1 s=CLOSED pgs=9957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:41.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:41.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:41.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:41.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c840 con 0x7f21cc005e30 2021-03-09T10:12:41.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0422c0 con 0x7f21c4003cb0 2021-03-09T10:12:41.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085090 con 0x7f21cc0138f0 2021-03-09T10:12:41.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:41.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:41.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:41.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:41.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.130+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:41.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:41.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:41.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:41.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:41.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:12:41.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:12:41.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9967 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000bf80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:41.140 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:41.134+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33936 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:41.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc0138f0 2021-03-09T10:12:41.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fc70 con 0x7f21cc0138f0 2021-03-09T10:12:41.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:12:41.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.142+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:41.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.142+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:41.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:41.142+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:46.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:46.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:46.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9967 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000bf80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:46.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9967 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000bf80).stop 2021-03-09T10:12:46.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:46.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=9967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:46.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:46.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:46.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:46.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ef0 con 0x7f21cc005e30 2021-03-09T10:12:46.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042460 con 0x7f21c4003cb0 2021-03-09T10:12:46.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091820 con 0x7f21cc042f20 2021-03-09T10:12:46.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:46.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:46.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:46.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:46.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:46.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:46.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:46.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:46.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:46.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091960 con 0x7f21cc042f20 2021-03-09T10:12:46.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.138+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc045bd0 con 0x7f21cc042f20 2021-03-09T10:12:46.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 secure :-1 s=READY pgs=9976 cs=0 l=1 rev1=1 rx=0x7f21e4038960 tx=0x7f21e40081c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:46.147 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:46.142+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33975 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:46.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401f370 con 0x7f21cc042f20 2021-03-09T10:12:46.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4038990 con 0x7f21cc042f20 2021-03-09T10:12:46.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc042f20 2021-03-09T10:12:46.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.150+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:46.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.150+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:46.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:46.150+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:51.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:51.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:51.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 secure :-1 s=READY pgs=9976 cs=0 l=1 rev1=1 rx=0x7f21e4038960 tx=0x7f21e40081c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:51.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 secure :-1 s=READY pgs=9976 cs=0 l=1 rev1=1 rx=0x7f21e4038960 tx=0x7f21e40081c0).stop 2021-03-09T10:12:51.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02fab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:51.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02fab0 unknown :-1 s=CLOSED pgs=9976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:51.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:51.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:51.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:51.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013090 con 0x7f21cc005e30 2021-03-09T10:12:51.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0917d0 con 0x7f21c4003cb0 2021-03-09T10:12:51.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045bd0 con 0x7f21cc0138f0 2021-03-09T10:12:51.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:51.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:51.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:51.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:51.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:51.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:51.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:51.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:51.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:51.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc045d10 con 0x7f21cc0138f0 2021-03-09T10:12:51.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:12:51.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.146+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9988 cs=0 l=1 rev1=1 rx=0x7f21e001b010 tx=0x7f21e00a6270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:51.154 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:51.150+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34011 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:51.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21cc0138f0 2021-03-09T10:12:51.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003a000 con 0x7f21cc0138f0 2021-03-09T10:12:51.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:12:51.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.154+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:51.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:51.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:51.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:12:54.984 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:54.994 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:12:56.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:56.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:56.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9988 cs=0 l=1 rev1=1 rx=0x7f21e001b010 tx=0x7f21e00a6270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:56.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=9988 cs=0 l=1 rev1=1 rx=0x7f21e001b010 tx=0x7f21e00a6270).stop 2021-03-09T10:12:56.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:56.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=9988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:56.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:56.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:56.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:56.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045bd0 con 0x7f21cc005e30 2021-03-09T10:12:56.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013090 con 0x7f21c4003cb0 2021-03-09T10:12:56.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012b80 con 0x7f21cc042f20 2021-03-09T10:12:56.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:12:56.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:56.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:12:56.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:56.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.150+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:56.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:56.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:56.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:56.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:56.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00c720 con 0x7f21cc042f20 2021-03-09T10:12:56.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ea80 con 0x7f21cc042f20 2021-03-09T10:12:56.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.154+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041f80 secure :-1 s=READY pgs=9998 cs=0 l=1 rev1=1 rx=0x7f21dc058ea0 tx=0x7f21dc081020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:56.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005e50 con 0x7f21cc042f20 2021-03-09T10:12:56.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc042f20 2021-03-09T10:12:56.162 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:12:56.158+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34050 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:12:56.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.162+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:12:56.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.162+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:12:56.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.162+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:12:56.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:12:56.162+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:01.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:01.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:01.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041f80 secure :-1 s=READY pgs=9998 cs=0 l=1 rev1=1 rx=0x7f21dc058ea0 tx=0x7f21dc081020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:01.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041f80 secure :-1 s=READY pgs=9998 cs=0 l=1 rev1=1 rx=0x7f21dc058ea0 tx=0x7f21dc081020).stop 2021-03-09T10:13:01.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:01.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041f80 unknown :-1 s=CLOSED pgs=9998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:01.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:01.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:01.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:01.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013090 con 0x7f21cc005e30 2021-03-09T10:13:01.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ea80 con 0x7f21c4003cb0 2021-03-09T10:13:01.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:13:01.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:01.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:01.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:01.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:01.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:01.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:01.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:01.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:01.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:01.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d0e0 con 0x7f21cc0138f0 2021-03-09T10:13:01.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00c720 con 0x7f21cc0138f0 2021-03-09T10:13:01.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.158+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10008 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc00a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:01.167 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:01.162+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34086 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:01.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc0138f0 2021-03-09T10:13:01.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc0138f0 2021-03-09T10:13:01.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.166+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052000 con 0x7f21cc0138f0 2021-03-09T10:13:01.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.166+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:01.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:01.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:01.166+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:06.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:06.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:06.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10008 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc00a870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:06.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10008 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc00a870).stop 2021-03-09T10:13:06.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:06.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:06.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:06.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:06.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:06.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013090 con 0x7f21cc005e30 2021-03-09T10:13:06.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d430 con 0x7f21c4003cb0 2021-03-09T10:13:06.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:13:06.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:06.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:06.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:06.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:06.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:06.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:06.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:06.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:06.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:06.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ea80 con 0x7f21cc042f20 2021-03-09T10:13:06.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d0e0 con 0x7f21cc042f20 2021-03-09T10:13:06.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10022 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0ab1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:06.175 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:06.169+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34125 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:06.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:13:06.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028bb0 con 0x7f21cc042f20 2021-03-09T10:13:06.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc042f20 2021-03-09T10:13:06.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.177+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:06.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:06.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:06.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:11.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:11.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:11.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10022 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0ab1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:11.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10022 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0ab1c0).stop 2021-03-09T10:13:11.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:11.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=CLOSED pgs=10022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:11.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:11.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:11.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:11.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:13:11.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ef0 con 0x7f21c4003cb0 2021-03-09T10:13:11.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:11.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013030 con 0x7f21cc0138f0 2021-03-09T10:13:11.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:11.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:11.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:11.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.173+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:11.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:11.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:11.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:11.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:11.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d400 con 0x7f21cc0138f0 2021-03-09T10:13:11.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc045d10 con 0x7f21cc0138f0 2021-03-09T10:13:11.182 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34161 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:11.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10035 cs=0 l=1 rev1=1 rx=0x7f21e000c010 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:11.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc0138f0 2021-03-09T10:13:11.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0138f0 2021-03-09T10:13:11.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:13:11.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.185+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:11.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:11.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:11.185+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:16.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:16.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:16.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10035 cs=0 l=1 rev1=1 rx=0x7f21e000c010 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:16.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10035 cs=0 l=1 rev1=1 rx=0x7f21e000c010 tx=0x7f21e000b3e0).stop 2021-03-09T10:13:16.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:16.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:16.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:16.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:16.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:16.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013090 con 0x7f21cc005e30 2021-03-09T10:13:16.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:13:16.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d0b0 con 0x7f21cc042f20 2021-03-09T10:13:16.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:16.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:16.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.177+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:16.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:16.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:16.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:16.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:16.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:16.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:16.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d400 con 0x7f21cc042f20 2021-03-09T10:13:16.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc042f20 2021-03-09T10:13:16.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.181+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10040 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0013e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:16.189 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:16.185+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34200 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:16.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21cc042f20 2021-03-09T10:13:16.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0007580 con 0x7f21cc042f20 2021-03-09T10:13:16.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:13:16.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.189+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:16.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:16.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:16.189+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:21.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:21.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:21.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10040 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0013e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:21.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10040 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0013e60).stop 2021-03-09T10:13:21.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:21.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=CLOSED pgs=10040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:21.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:21.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:21.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:21.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ed0 con 0x7f21cc005e30 2021-03-09T10:13:21.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013010 con 0x7f21c4003cb0 2021-03-09T10:13:21.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:13:21.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:21.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:21.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:21.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:21.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.185+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:21.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:21.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:21.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:21.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:21.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d0b0 con 0x7f21cc0138f0 2021-03-09T10:13:21.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ea80 con 0x7f21cc0138f0 2021-03-09T10:13:21.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10051 cs=0 l=1 rev1=1 rx=0x7f21cc01dda0 tx=0x7f21dc0ab150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:21.195 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34236 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:21.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018d10 con 0x7f21cc0138f0 2021-03-09T10:13:21.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b6960 con 0x7f21cc0138f0 2021-03-09T10:13:21.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00a8e0 con 0x7f21cc0138f0 2021-03-09T10:13:21.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.193+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:21.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.197+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:21.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:21.197+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:25.040 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:25.043 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:26.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:26.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:26.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10051 cs=0 l=1 rev1=1 rx=0x7f21cc01dda0 tx=0x7f21dc0ab150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:26.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10051 cs=0 l=1 rev1=1 rx=0x7f21cc01dda0 tx=0x7f21dc0ab150).stop 2021-03-09T10:13:26.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:26.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:26.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:26.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:26.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:26.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:13:26.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013070 con 0x7f21c4003cb0 2021-03-09T10:13:26.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0422b0 con 0x7f21cc042f20 2021-03-09T10:13:26.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:26.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:26.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:26.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:26.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:26.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:26.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:26.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:26.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:26.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0423f0 con 0x7f21cc042f20 2021-03-09T10:13:26.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d400 con 0x7f21cc042f20 2021-03-09T10:13:26.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.193+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10062 cs=0 l=1 rev1=1 rx=0x7f21e0076c30 tx=0x7f21e0013df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:26.202 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:26.197+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34273 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:26.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:13:26.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004b10 con 0x7f21cc042f20 2021-03-09T10:13:26.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:13:26.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.201+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:26.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:26.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:26.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:31.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:31.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:31.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10062 cs=0 l=1 rev1=1 rx=0x7f21e0076c30 tx=0x7f21e0013df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:31.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 secure :-1 s=READY pgs=10062 cs=0 l=1 rev1=1 rx=0x7f21e0076c30 tx=0x7f21e0013df0).stop 2021-03-09T10:13:31.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:31.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041dc0 unknown :-1 s=CLOSED pgs=10062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:31.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:31.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:31.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:31.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:13:31.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026400 con 0x7f21c4003cb0 2021-03-09T10:13:31.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d400 con 0x7f21cc0138f0 2021-03-09T10:13:31.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:31.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:31.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:31.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:31.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.197+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:31.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:31.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:31.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:31.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:31.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d540 con 0x7f21cc0138f0 2021-03-09T10:13:31.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21cc0138f0 2021-03-09T10:13:31.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.201+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10074 cs=0 l=1 rev1=1 rx=0x7f21e00077b0 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:31.209 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:31.205+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34311 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:31.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.205+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:13:31.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e009a8a0 con 0x7f21cc0138f0 2021-03-09T10:13:31.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009aa60 con 0x7f21cc0138f0 2021-03-09T10:13:31.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.213+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:31.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.213+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:31.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:31.213+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:36.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:36.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:36.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10074 cs=0 l=1 rev1=1 rx=0x7f21e00077b0 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:36.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10074 cs=0 l=1 rev1=1 rx=0x7f21e00077b0 tx=0x7f21e0003740).stop 2021-03-09T10:13:36.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:36.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:36.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:36.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:36.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:36.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:13:36.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056a00 con 0x7f21c4003cb0 2021-03-09T10:13:36.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042240 con 0x7f21cc042f20 2021-03-09T10:13:36.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:36.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:36.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:36.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:36.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:36.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:36.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:36.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:36.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:36.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0423b0 con 0x7f21cc042f20 2021-03-09T10:13:36.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d0b0 con 0x7f21cc042f20 2021-03-09T10:13:36.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041d50 secure :-1 s=READY pgs=10083 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc004020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:36.217 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:36.213+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34347 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:36.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc042f20 2021-03-09T10:13:36.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc042f20 2021-03-09T10:13:36.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc042f20 2021-03-09T10:13:36.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.217+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:36.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.217+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:36.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:36.217+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:41.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:41.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:41.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041d50 secure :-1 s=READY pgs=10083 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc004020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:41.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041d50 secure :-1 s=READY pgs=10083 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc004020).stop 2021-03-09T10:13:41.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:41.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041d50 unknown :-1 s=CLOSED pgs=10083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:41.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:41.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:41.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:41.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042240 con 0x7f21cc005e30 2021-03-09T10:13:41.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0569a0 con 0x7f21c4003cb0 2021-03-09T10:13:41.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ae0 con 0x7f21cc0138f0 2021-03-09T10:13:41.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:41.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:41.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:41.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:41.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:41.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:41.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:41.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:41.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:41.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:13:41.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.213+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d540 con 0x7f21cc0138f0 2021-03-09T10:13:41.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.217+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10095 cs=0 l=1 rev1=1 rx=0x7f21e0084200 tx=0x7f21e007d210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:41.223 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:41.217+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34386 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:41.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e002bd20 con 0x7f21cc0138f0 2021-03-09T10:13:41.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.225+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:13:41.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091d60 con 0x7f21cc0138f0 2021-03-09T10:13:41.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.229+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:41.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.229+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:41.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:41.229+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:46.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:46.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:46.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10095 cs=0 l=1 rev1=1 rx=0x7f21e0084200 tx=0x7f21e007d210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:46.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10095 cs=0 l=1 rev1=1 rx=0x7f21e0084200 tx=0x7f21e007d210).stop 2021-03-09T10:13:46.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:46.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:46.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:46.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:46.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:46.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d100 con 0x7f21cc005e30 2021-03-09T10:13:46.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0421a0 con 0x7f21c4003cb0 2021-03-09T10:13:46.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012630 con 0x7f21cc042f20 2021-03-09T10:13:46.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:46.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:46.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:46.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:46.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:46.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:46.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:46.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:46.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:46.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012770 con 0x7f21cc042f20 2021-03-09T10:13:46.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.221+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0423b0 con 0x7f21cc042f20 2021-03-09T10:13:46.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.225+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041cb0 secure :-1 s=READY pgs=10104 cs=0 l=1 rev1=1 rx=0x7f21dc06c040 tx=0x7f21dc0800b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:46.230 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:46.225+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34422 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:46.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ba470 con 0x7f21cc042f20 2021-03-09T10:13:46.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005ed0 con 0x7f21cc042f20 2021-03-09T10:13:46.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc050020 con 0x7f21cc042f20 2021-03-09T10:13:46.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.233+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:46.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.237+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:46.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:46.237+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:51.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:13:51.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:51.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041cb0 secure :-1 s=READY pgs=10104 cs=0 l=1 rev1=1 rx=0x7f21dc06c040 tx=0x7f21dc0800b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:51.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041cb0 secure :-1 s=READY pgs=10104 cs=0 l=1 rev1=1 rx=0x7f21dc06c040 tx=0x7f21dc0800b0).stop 2021-03-09T10:13:51.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:51.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041cb0 unknown :-1 s=CLOSED pgs=10104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:51.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:51.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:51.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:51.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012900 con 0x7f21cc005e30 2021-03-09T10:13:51.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012610 con 0x7f21c4003cb0 2021-03-09T10:13:51.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d100 con 0x7f21cc0138f0 2021-03-09T10:13:51.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:51.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:51.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:51.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:51.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:51.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:51.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:51.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:51.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:51.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0421a0 con 0x7f21cc0138f0 2021-03-09T10:13:51.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012770 con 0x7f21cc0138f0 2021-03-09T10:13:51.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.228+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10115 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0502d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:51.240 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:51.232+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34461 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:51.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005ed0 con 0x7f21cc0138f0 2021-03-09T10:13:51.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ba470 con 0x7f21cc0138f0 2021-03-09T10:13:51.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21cc0138f0 2021-03-09T10:13:51.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.240+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:51.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.240+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:51.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:51.240+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:13:55.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:55.105 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:13:56.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:13:56.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:56.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10115 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0502d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:56.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10115 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0502d0).stop 2021-03-09T10:13:56.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:56.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:56.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:56.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:56.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:56.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042330 con 0x7f21cc005e30 2021-03-09T10:13:56.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042470 con 0x7f21c4003cb0 2021-03-09T10:13:56.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0125c0 con 0x7f21cc042f20 2021-03-09T10:13:56.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:13:56.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:56.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:13:56.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:56.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:56.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:56.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:56.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:56.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:56.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012700 con 0x7f21cc042f20 2021-03-09T10:13:56.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.236+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091960 con 0x7f21cc042f20 2021-03-09T10:13:56.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041c10 secure :-1 s=READY pgs=10126 cs=0 l=1 rev1=1 rx=0x7f21e40b7040 tx=0x7f21e4091190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:56.246 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:13:56.240+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34496 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:13:56.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:13:56.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:13:56.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.244+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:13:56.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.244+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:13:56.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.248+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:13:56.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:13:56.248+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:01.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:01.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:01.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041c10 secure :-1 s=READY pgs=10126 cs=0 l=1 rev1=1 rx=0x7f21e40b7040 tx=0x7f21e4091190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:01.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041c10 secure :-1 s=READY pgs=10126 cs=0 l=1 rev1=1 rx=0x7f21e40b7040 tx=0x7f21e4091190).stop 2021-03-09T10:14:01.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:01.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041c10 unknown :-1 s=CLOSED pgs=10126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:01.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:01.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:01.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:01.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042330 con 0x7f21cc005e30 2021-03-09T10:14:01.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042470 con 0x7f21c4003cb0 2021-03-09T10:14:01.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.240+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048ed0 con 0x7f21cc0138f0 2021-03-09T10:14:01.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:01.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:01.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:01.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:01.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:01.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:01.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:01.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:01.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:01.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049010 con 0x7f21cc0138f0 2021-03-09T10:14:01.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:14:01.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.244+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7f21e00a8cb0 tx=0x7f21e007d620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:01.254 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:01.248+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34536 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:01.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc0138f0 2021-03-09T10:14:01.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:14:01.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:14:01.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.252+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:01.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.252+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:01.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:01.252+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:06.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:06.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:06.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7f21e00a8cb0 tx=0x7f21e007d620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:06.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7f21e00a8cb0 tx=0x7f21e007d620).stop 2021-03-09T10:14:06.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:06.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:06.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:06.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:06.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:06.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fb60 con 0x7f21cc005e30 2021-03-09T10:14:06.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042060 con 0x7f21c4003cb0 2021-03-09T10:14:06.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048eb0 con 0x7f21cc042f20 2021-03-09T10:14:06.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:06.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:06.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:06.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:06.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:06.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:06.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:06.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:06.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:06.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:14:06.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012700 con 0x7f21cc042f20 2021-03-09T10:14:06.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.252+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041b70 secure :-1 s=READY pgs=10147 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:06.261 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:06.256+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34571 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:06.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:14:06.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:14:06.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T10:14:06.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.260+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:06.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.260+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:06.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:06.260+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:11.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:11.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:11.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041b70 secure :-1 s=READY pgs=10147 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:11.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041b70 secure :-1 s=READY pgs=10147 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4005180).stop 2021-03-09T10:14:11.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:11.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041b70 unknown :-1 s=CLOSED pgs=10147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:11.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:11.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:11.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:11.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042060 con 0x7f21cc005e30 2021-03-09T10:14:11.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048ea0 con 0x7f21c4003cb0 2021-03-09T10:14:11.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042330 con 0x7f21cc0138f0 2021-03-09T10:14:11.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:11.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:11.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:11.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:11.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:11.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:11.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:11.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:11.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:11.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042470 con 0x7f21cc0138f0 2021-03-09T10:14:11.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0421a0 con 0x7f21cc0138f0 2021-03-09T10:14:11.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10158 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc015a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:11.267 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:11.260+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34611 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:11.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc032a80 con 0x7f21cc0138f0 2021-03-09T10:14:11.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069120 con 0x7f21cc0138f0 2021-03-09T10:14:11.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc050020 con 0x7f21cc0138f0 2021-03-09T10:14:11.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.268+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:11.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.268+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:11.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:11.268+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:16.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:16.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:16.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10158 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc015a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:16.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10158 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc015a20).stop 2021-03-09T10:14:16.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:16.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:16.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:16.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:16.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:16.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:16.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048ea0 con 0x7f21cc005e30 2021-03-09T10:14:16.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.260+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:16.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041fc0 con 0x7f21c4003cb0 2021-03-09T10:14:16.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042100 con 0x7f21cc042f20 2021-03-09T10:14:16.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:16.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:16.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:16.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:16.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:16.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:16.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:16.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042270 con 0x7f21cc042f20 2021-03-09T10:14:16.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:14:16.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.264+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041ad0 secure :-1 s=READY pgs=10168 cs=0 l=1 rev1=1 rx=0x7f21e404f640 tx=0x7f21e40362e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:16.274 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:16.268+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34646 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:16.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d1c0 con 0x7f21cc042f20 2021-03-09T10:14:16.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:14:16.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:14:16.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.272+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:16.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.272+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:16.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:16.272+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:21.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:21.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:21.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041ad0 secure :-1 s=READY pgs=10168 cs=0 l=1 rev1=1 rx=0x7f21e404f640 tx=0x7f21e40362e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:21.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041ad0 secure :-1 s=READY pgs=10168 cs=0 l=1 rev1=1 rx=0x7f21e404f640 tx=0x7f21e40362e0).stop 2021-03-09T10:14:21.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:21.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041ad0 unknown :-1 s=CLOSED pgs=10168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:21.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:21.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:21.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:21.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048ea0 con 0x7f21cc005e30 2021-03-09T10:14:21.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0539e0 con 0x7f21c4003cb0 2021-03-09T10:14:21.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ec0 con 0x7f21cc0138f0 2021-03-09T10:14:21.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:21.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:21.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:21.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:21.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:21.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:21.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:21.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:21.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:21.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:14:21.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042470 con 0x7f21cc0138f0 2021-03-09T10:14:21.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.272+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10181 cs=0 l=1 rev1=1 rx=0x7f21cc040130 tx=0x7f21dc0098f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:21.282 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:21.276+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34686 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:21.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:14:21.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21cc0138f0 2021-03-09T10:14:21.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052df0 con 0x7f21cc0138f0 2021-03-09T10:14:21.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.284+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:21.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.284+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:21.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:21.284+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:25.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:25.164 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:26.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:14:26.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:26.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10181 cs=0 l=1 rev1=1 rx=0x7f21cc040130 tx=0x7f21dc0098f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:26.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10181 cs=0 l=1 rev1=1 rx=0x7f21cc040130 tx=0x7f21dc0098f0).stop 2021-03-09T10:14:26.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:26.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:26.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:26.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:26.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:26.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048ea0 con 0x7f21cc005e30 2021-03-09T10:14:26.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0427b0 con 0x7f21c4003cb0 2021-03-09T10:14:26.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041e00 con 0x7f21cc042f20 2021-03-09T10:14:26.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:26.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:26.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:26.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:26.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:26.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:26.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:26.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:26.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:26.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041f40 con 0x7f21cc042f20 2021-03-09T10:14:26.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049010 con 0x7f21cc042f20 2021-03-09T10:14:26.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041910 secure :-1 s=READY pgs=10192 cs=0 l=1 rev1=1 rx=0x7f21e0091000 tx=0x7f21e000b4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:26.288 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:26.284+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34721 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:26.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc042f20 2021-03-09T10:14:26.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc042f20 2021-03-09T10:14:26.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:14:26.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.292+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:26.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:26.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:26.292+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:31.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:31.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:31.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041910 secure :-1 s=READY pgs=10192 cs=0 l=1 rev1=1 rx=0x7f21e0091000 tx=0x7f21e000b4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:31.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041910 secure :-1 s=READY pgs=10192 cs=0 l=1 rev1=1 rx=0x7f21e0091000 tx=0x7f21e000b4d0).stop 2021-03-09T10:14:31.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc041910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:31.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc041910 unknown :-1 s=CLOSED pgs=10192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:31.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:31.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:31.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:31.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041e00 con 0x7f21cc005e30 2021-03-09T10:14:31.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc048ea0 con 0x7f21c4003cb0 2021-03-09T10:14:31.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f380 con 0x7f21cc0138f0 2021-03-09T10:14:31.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:31.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:31.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:31.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:31.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:31.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:31.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:31.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:31.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:31.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0427b0 con 0x7f21cc0138f0 2021-03-09T10:14:31.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042270 con 0x7f21cc0138f0 2021-03-09T10:14:31.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10200 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e40074d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:31.296 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34761 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:31.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404e0e0 con 0x7f21cc0138f0 2021-03-09T10:14:31.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404e0e0 con 0x7f21cc0138f0 2021-03-09T10:14:31.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:14:31.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.292+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:31.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.296+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:31.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:31.296+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:36.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:14:36.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:36.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10200 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e40074d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:36.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10200 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e40074d0).stop 2021-03-09T10:14:36.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:36.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:36.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:36.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:36.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:36.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04bdc0 con 0x7f21cc005e30 2021-03-09T10:14:36.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0280e0 con 0x7f21c4003cb0 2021-03-09T10:14:36.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030970 con 0x7f21cc042f20 2021-03-09T10:14:36.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:36.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:36.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:36.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:36.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:36.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:36.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:36.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:36.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:36.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041db0 con 0x7f21cc042f20 2021-03-09T10:14:36.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.291+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041f40 con 0x7f21cc042f20 2021-03-09T10:14:36.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.295+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0418c0 secure :-1 s=READY pgs=10211 cs=0 l=1 rev1=1 rx=0x7f21e0003860 tx=0x7f21e009d4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:36.304 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:36.295+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34796 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:36.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:14:36.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc042f20 2021-03-09T10:14:36.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009dd80 con 0x7f21cc042f20 2021-03-09T10:14:36.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.299+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:36.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:36.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:36.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:41.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:41.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:41.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0418c0 secure :-1 s=READY pgs=10211 cs=0 l=1 rev1=1 rx=0x7f21e0003860 tx=0x7f21e009d4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:41.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0418c0 secure :-1 s=READY pgs=10211 cs=0 l=1 rev1=1 rx=0x7f21e0003860 tx=0x7f21e009d4e0).stop 2021-03-09T10:14:41.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0418c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:41.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0418c0 unknown :-1 s=CLOSED pgs=10211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:41.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:41.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:41.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:41.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0280e0 con 0x7f21cc005e30 2021-03-09T10:14:41.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041f70 con 0x7f21c4003cb0 2021-03-09T10:14:41.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02fdf0 con 0x7f21cc0138f0 2021-03-09T10:14:41.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:41.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:41.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:41.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:41.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:41.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:41.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:41.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:41.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:41.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089500 con 0x7f21cc0138f0 2021-03-09T10:14:41.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:14:41.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.299+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10222 cs=0 l=1 rev1=1 rx=0x7f21e0059ff0 tx=0x7f21e009d880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:41.310 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:41.303+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34836 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:41.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc0138f0 2021-03-09T10:14:41.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e009de50 con 0x7f21cc0138f0 2021-03-09T10:14:41.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:14:41.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.307+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:41.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.307+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:41.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:41.307+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:46.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:46.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:46.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10222 cs=0 l=1 rev1=1 rx=0x7f21e0059ff0 tx=0x7f21e009d880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:46.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10222 cs=0 l=1 rev1=1 rx=0x7f21e0059ff0 tx=0x7f21e009d880).stop 2021-03-09T10:14:46.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:46.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:46.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:46.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:46.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:46.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc005e30 2021-03-09T10:14:46.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041be0 con 0x7f21c4003cb0 2021-03-09T10:14:46.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041d20 con 0x7f21cc042f20 2021-03-09T10:14:46.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:46.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:46.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:46.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:46.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.307+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:46.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:46.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:46.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:46.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:46.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041e60 con 0x7f21cc042f20 2021-03-09T10:14:46.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:14:46.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.311+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0416f0 secure :-1 s=READY pgs=10233 cs=0 l=1 rev1=1 rx=0x7f21cc0404a0 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:46.316 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:46.315+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34871 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:46.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089000 con 0x7f21cc042f20 2021-03-09T10:14:46.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089000 con 0x7f21cc042f20 2021-03-09T10:14:46.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.319+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:14:46.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.319+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:46.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:46.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:46.323+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:51.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:51.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:51.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0416f0 secure :-1 s=READY pgs=10233 cs=0 l=1 rev1=1 rx=0x7f21cc0404a0 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:51.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0416f0 secure :-1 s=READY pgs=10233 cs=0 l=1 rev1=1 rx=0x7f21cc0404a0 tx=0x7f21dc010d90).stop 2021-03-09T10:14:51.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0416f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:51.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0416f0 unknown :-1 s=CLOSED pgs=10233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:51.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:51.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:51.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:51.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013270 con 0x7f21cc005e30 2021-03-09T10:14:51.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043560 con 0x7f21c4003cb0 2021-03-09T10:14:51.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:14:51.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:51.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:51.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:51.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:51.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.315+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:51.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:51.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:51.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:51.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:51.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:14:51.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0427b0 con 0x7f21cc0138f0 2021-03-09T10:14:51.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.319+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10246 cs=0 l=1 rev1=1 rx=0x7f21e402b7d0 tx=0x7f21e402b6f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:51.324 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:51.323+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34911 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:51.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc0138f0 2021-03-09T10:14:51.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc0138f0 2021-03-09T10:14:51.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b5220 con 0x7f21cc0138f0 2021-03-09T10:14:51.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.327+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:51.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.327+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:51.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:51.327+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:14:55.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:55.215 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:14:56.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:56.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:56.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10246 cs=0 l=1 rev1=1 rx=0x7f21e402b7d0 tx=0x7f21e402b6f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:56.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 secure :-1 s=READY pgs=10246 cs=0 l=1 rev1=1 rx=0x7f21e402b7d0 tx=0x7f21e402b6f0).stop 2021-03-09T10:14:56.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc08d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:56.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc08d670 unknown :-1 s=CLOSED pgs=10246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:56.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:56.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:56.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:56.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043700 con 0x7f21cc005e30 2021-03-09T10:14:56.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041c50 con 0x7f21c4003cb0 2021-03-09T10:14:56.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:14:56.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:14:56.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:14:56.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:56.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:56.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:56.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:56.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:56.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:56.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:56.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b920 con 0x7f21cc042f20 2021-03-09T10:14:56.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.323+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:14:56.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cf0 secure :-1 s=READY pgs=10255 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e40b54f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:56.332 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:14:56.327+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34946 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:14:56.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.331+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21cc042f20 2021-03-09T10:14:56.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:14:56.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21cc042f20 2021-03-09T10:14:56.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.335+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:14:56.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:14:56.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:14:56.335+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:01.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:01.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:01.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cf0 secure :-1 s=READY pgs=10255 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e40b54f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:01.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cf0 secure :-1 s=READY pgs=10255 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e40b54f0).stop 2021-03-09T10:15:01.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:01.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cf0 unknown :-1 s=CLOSED pgs=10255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:01.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:01.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:01.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0851e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:01.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:01.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cb10 con 0x7f21c4003cb0 2021-03-09T10:15:01.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:15:01.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.327+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:01.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:01.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:01.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:01.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0851e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:01.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:01.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:01.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:01.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:01.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:15:01.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.331+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041e60 con 0x7f21cc0138f0 2021-03-09T10:15:01.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0851e0 secure :-1 s=READY pgs=10267 cs=0 l=1 rev1=1 rx=0x7f21dc02d060 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:01.337 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:01.335+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34986 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:01.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.339+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06d650 con 0x7f21cc0138f0 2021-03-09T10:15:01.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06d880 con 0x7f21cc0138f0 2021-03-09T10:15:01.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:15:01.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.343+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:01.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.347+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:01.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:01.347+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:06.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0851e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:15:06.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0851e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:06.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0851e0 secure :-1 s=READY pgs=10267 cs=0 l=1 rev1=1 rx=0x7f21dc02d060 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:06.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0851e0 secure :-1 s=READY pgs=10267 cs=0 l=1 rev1=1 rx=0x7f21dc02d060 tx=0x7f21dc02dfc0).stop 2021-03-09T10:15:06.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0851e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:06.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0851e0 unknown :-1 s=CLOSED pgs=10267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:06.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:06.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:06.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:06.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:06.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01aed0 con 0x7f21c4003cb0 2021-03-09T10:15:06.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.335+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:15:06.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.339+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:06.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.339+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:06.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.339+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:06.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.339+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:06.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.339+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:06.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:06.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:06.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:06.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:06.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013410 con 0x7f21cc042f20 2021-03-09T10:15:06.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089500 con 0x7f21cc042f20 2021-03-09T10:15:06.345 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35021 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:06.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10281 cs=0 l=1 rev1=1 rx=0x7f21e0037aa0 tx=0x7f21e009d790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:06.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21cc042f20 2021-03-09T10:15:06.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21cc042f20 2021-03-09T10:15:06.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:15:06.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.351+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:06.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.351+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:06.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:06.351+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:11.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:11.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:11.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10281 cs=0 l=1 rev1=1 rx=0x7f21e0037aa0 tx=0x7f21e009d790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:11.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10281 cs=0 l=1 rev1=1 rx=0x7f21e0037aa0 tx=0x7f21e009d790).stop 2021-03-09T10:15:11.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:11.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:11.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:11.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:11.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:11.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:11.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b070 con 0x7f21c4003cb0 2021-03-09T10:15:11.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ead0 con 0x7f21cc0138f0 2021-03-09T10:15:11.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:11.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:11.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:11.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:11.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.343+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:11.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:11.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:11.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:11.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:11.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041e60 con 0x7f21cc0138f0 2021-03-09T10:15:11.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013410 con 0x7f21cc0138f0 2021-03-09T10:15:11.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.347+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042450 secure :-1 s=READY pgs=10286 cs=0 l=1 rev1=1 rx=0x7f21e00a97d0 tx=0x7f21e0007f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:11.353 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:11.351+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35061 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:11.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc0138f0 2021-03-09T10:15:11.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:15:11.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.355+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:15:11.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.355+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:11.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.355+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:11.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:11.355+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:16.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:15:16.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:16.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042450 secure :-1 s=READY pgs=10286 cs=0 l=1 rev1=1 rx=0x7f21e00a97d0 tx=0x7f21e0007f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:16.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042450 secure :-1 s=READY pgs=10286 cs=0 l=1 rev1=1 rx=0x7f21e00a97d0 tx=0x7f21e0007f40).stop 2021-03-09T10:15:16.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:16.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042450 unknown :-1 s=CLOSED pgs=10286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:16.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:16.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:16.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:16.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:16.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:15:16.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041e60 con 0x7f21cc042f20 2021-03-09T10:15:16.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:16.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:16.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:16.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:16.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:16.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:16.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:16.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:16.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:16.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0896a0 con 0x7f21cc042f20 2021-03-09T10:15:16.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21cc042f20 2021-03-09T10:15:16.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.355+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10297 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:16.359 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:16.355+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35096 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:16.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:15:16.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.363+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d1d0 con 0x7f21cc042f20 2021-03-09T10:15:16.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.363+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21cc042f20 2021-03-09T10:15:16.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.363+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:16.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.363+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:16.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:16.363+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:21.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:21.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:21.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10297 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0004c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:21.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10297 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0004c50).stop 2021-03-09T10:15:21.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:21.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:21.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:21.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:21.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:21.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21cc005e30 2021-03-09T10:15:21.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:15:21.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041e60 con 0x7f21cc0138f0 2021-03-09T10:15:21.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:21.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:21.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:21.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:21.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:21.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:21.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:21.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:21.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:21.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b210 con 0x7f21cc0138f0 2021-03-09T10:15:21.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:15:21.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.359+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096ef0 secure :-1 s=READY pgs=10307 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc081090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:21.366 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:21.363+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35136 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:21.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.363+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00aa00 con 0x7f21cc0138f0 2021-03-09T10:15:21.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.363+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00ab60 con 0x7f21cc0138f0 2021-03-09T10:15:21.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.367+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc0138f0 2021-03-09T10:15:21.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.367+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:21.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.371+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:21.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:21.371+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:25.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:25.265 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:26.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc096ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:26.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc096ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:26.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096ef0 secure :-1 s=READY pgs=10307 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc081090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:26.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096ef0 secure :-1 s=READY pgs=10307 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc081090).stop 2021-03-09T10:15:26.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc096ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:26.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc096ef0 unknown :-1 s=CLOSED pgs=10307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:26.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:26.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:26.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:26.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21cc005e30 2021-03-09T10:15:26.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:15:26.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041e60 con 0x7f21cc042f20 2021-03-09T10:15:26.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:26.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:26.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:26.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:26.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:26.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:26.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.366+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:26.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:26.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.366+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:26.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:15:26.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0896a0 con 0x7f21cc042f20 2021-03-09T10:15:26.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.370+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10321 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e001f950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:26.373 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:26.370+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35171 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:26.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e001fad0 con 0x7f21cc042f20 2021-03-09T10:15:26.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0076db0 con 0x7f21cc042f20 2021-03-09T10:15:26.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0007c70 con 0x7f21cc042f20 2021-03-09T10:15:26.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.374+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:26.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.374+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:26.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:26.374+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:31.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:31.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:31.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10321 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e001f950).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:31.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10321 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e001f950).stop 2021-03-09T10:15:31.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:31.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:31.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:31.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:31.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:31.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:31.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21c4003cb0 2021-03-09T10:15:31.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:31.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:15:31.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:31.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:31.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:31.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:31.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:31.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:31.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:31.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:31.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041e60 con 0x7f21cc0138f0 2021-03-09T10:15:31.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b210 con 0x7f21cc0138f0 2021-03-09T10:15:31.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10328 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc078880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:31.380 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:31.374+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35211 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:31.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc078d80 con 0x7f21cc0138f0 2021-03-09T10:15:31.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0535a0 con 0x7f21cc0138f0 2021-03-09T10:15:31.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc078d80 con 0x7f21cc0138f0 2021-03-09T10:15:31.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.382+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:31.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.382+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:31.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:31.382+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:36.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:36.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:36.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10328 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc078880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:36.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10328 cs=0 l=1 rev1=1 rx=0x7f21dc077ed0 tx=0x7f21dc078880).stop 2021-03-09T10:15:36.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:36.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=CLOSED pgs=10328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:36.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:36.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:36.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:36.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21cc005e30 2021-03-09T10:15:36.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:36.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:15:36.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c9a0 con 0x7f21cc042f20 2021-03-09T10:15:36.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:36.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:36.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:36.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:36.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:36.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:36.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:36.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:36.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0896a0 con 0x7f21cc042f20 2021-03-09T10:15:36.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b920 con 0x7f21cc042f20 2021-03-09T10:15:36.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.378+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10339 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4024410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:36.386 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:36.382+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35246 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:36.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f230 con 0x7f21cc042f20 2021-03-09T10:15:36.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T10:15:36.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40240d0 con 0x7f21cc042f20 2021-03-09T10:15:36.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.386+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:36.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:36.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:36.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:41.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:41.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:41.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10339 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4024410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:41.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10339 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4024410).stop 2021-03-09T10:15:41.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:41.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:41.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:41.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:41.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:41.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:41.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21c4003cb0 2021-03-09T10:15:41.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c9a0 con 0x7f21cc0138f0 2021-03-09T10:15:41.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:41.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:41.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:41.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:41.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:41.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:41.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:41.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:41.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:41.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:15:41.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.386+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041e60 con 0x7f21cc0138f0 2021-03-09T10:15:41.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.390+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10353 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc080040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:41.394 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:41.390+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35286 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:41.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.394+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052c60 con 0x7f21cc0138f0 2021-03-09T10:15:41.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052e30 con 0x7f21cc0138f0 2021-03-09T10:15:41.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:15:41.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.398+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:41.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.398+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:41.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:41.398+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:46.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:46.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:46.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10353 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc080040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:46.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10353 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc080040).stop 2021-03-09T10:15:46.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:46.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=CLOSED pgs=10353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:46.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:46.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:46.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:46.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21cc005e30 2021-03-09T10:15:46.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:15:46.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:46.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c9a0 con 0x7f21cc042f20 2021-03-09T10:15:46.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.390+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:46.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:46.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:46.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:46.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:46.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:46.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:46.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:46.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041e60 con 0x7f21cc042f20 2021-03-09T10:15:46.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.394+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0896a0 con 0x7f21cc042f20 2021-03-09T10:15:46.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.398+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10362 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4059070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:46.402 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:46.398+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35321 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:46.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007230 con 0x7f21cc042f20 2021-03-09T10:15:46.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b5220 con 0x7f21cc042f20 2021-03-09T10:15:46.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.402+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc042f20 2021-03-09T10:15:46.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.402+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:46.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.402+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:46.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:46.402+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:51.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:51.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:51.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10362 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4059070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:51.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10362 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4059070).stop 2021-03-09T10:15:51.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:51.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:51.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:51.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:51.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:51.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21cc005e30 2021-03-09T10:15:51.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:15:51.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03c9a0 con 0x7f21cc0138f0 2021-03-09T10:15:51.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:51.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:51.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:51.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:51.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.398+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:51.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:51.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:51.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:51.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:51.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096b70 con 0x7f21cc0138f0 2021-03-09T10:15:51.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:15:51.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.402+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10371 cs=0 l=1 rev1=1 rx=0x7f21e0077060 tx=0x7f21e007d5c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:51.409 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:51.406+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35360 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:51.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.406+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc0138f0 2021-03-09T10:15:51.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.406+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:15:51.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.410+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:15:51.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.410+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:51.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.410+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:51.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:51.410+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:15:55.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:55.318 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:15:56.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:56.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:56.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10371 cs=0 l=1 rev1=1 rx=0x7f21e0077060 tx=0x7f21e007d5c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:56.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 secure :-1 s=READY pgs=10371 cs=0 l=1 rev1=1 rx=0x7f21e0077060 tx=0x7f21e007d5c0).stop 2021-03-09T10:15:56.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:56.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085190 unknown :-1 s=CLOSED pgs=10371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:56.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:56.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:56.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:56.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:15:56.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0566f0 con 0x7f21c4003cb0 2021-03-09T10:15:56.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052fc0 con 0x7f21cc042f20 2021-03-09T10:15:56.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:15:56.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:56.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:15:56.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:56.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:56.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:56.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:56.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:56.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.406+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:56.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.410+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0552b0 con 0x7f21cc042f20 2021-03-09T10:15:56.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.410+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041e60 con 0x7f21cc042f20 2021-03-09T10:15:56.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.410+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10383 cs=0 l=1 rev1=1 rx=0x7f21cc040db0 tx=0x7f21e40082c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:56.415 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:15:56.414+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35397 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:15:56.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.414+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4008620 con 0x7f21cc042f20 2021-03-09T10:15:56.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.414+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40052d0 con 0x7f21cc042f20 2021-03-09T10:15:56.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.418+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5d40 con 0x7f21cc042f20 2021-03-09T10:15:56.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.418+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:15:56.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.418+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:15:56.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:15:56.418+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:01.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:01.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:01.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10383 cs=0 l=1 rev1=1 rx=0x7f21cc040db0 tx=0x7f21e40082c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:01.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 secure :-1 s=READY pgs=10383 cs=0 l=1 rev1=1 rx=0x7f21cc040db0 tx=0x7f21e40082c0).stop 2021-03-09T10:16:01.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:01.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:01.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:01.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:01.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:01.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:16:01.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:16:01.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052fc0 con 0x7f21cc0138f0 2021-03-09T10:16:01.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:01.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:01.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:01.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:01.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:01.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:01.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:01.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:01.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:01.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0263e0 con 0x7f21cc0138f0 2021-03-09T10:16:01.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.414+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096b70 con 0x7f21cc0138f0 2021-03-09T10:16:01.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.418+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02cfc0 secure :-1 s=READY pgs=10393 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0014e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:01.423 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:01.418+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35435 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:01.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc0138f0 2021-03-09T10:16:01.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.426+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:16:01.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:16:01.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.430+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:01.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.430+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:01.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:01.430+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:06.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:06.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:06.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02cfc0 secure :-1 s=READY pgs=10393 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0014e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:06.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02cfc0 secure :-1 s=READY pgs=10393 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0014e90).stop 2021-03-09T10:16:06.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02cfc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:06.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02cfc0 unknown :-1 s=CLOSED pgs=10393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:06.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:06.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:06.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:06.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d650 con 0x7f21cc005e30 2021-03-09T10:16:06.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:16:06.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:16:06.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:06.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:06.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:06.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:06.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.422+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:06.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:06.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:06.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:06.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:06.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096d10 con 0x7f21cc042f20 2021-03-09T10:16:06.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:16:06.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046220 secure :-1 s=READY pgs=10410 cs=0 l=1 rev1=1 rx=0x7f21dc01a040 tx=0x7f21dc015a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:06.430 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:06.426+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35472 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:06.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.430+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:16:06.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:16:06.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.434+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21cc042f20 2021-03-09T10:16:06.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.434+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:06.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.438+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:06.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:06.438+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:11.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:11.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:11.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046220 secure :-1 s=READY pgs=10410 cs=0 l=1 rev1=1 rx=0x7f21dc01a040 tx=0x7f21dc015a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:11.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046220 secure :-1 s=READY pgs=10410 cs=0 l=1 rev1=1 rx=0x7f21dc01a040 tx=0x7f21dc015a90).stop 2021-03-09T10:16:11.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:11.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046220 unknown :-1 s=CLOSED pgs=10410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:11.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:11.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:11.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:11.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:16:11.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21c4003cb0 2021-03-09T10:16:11.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ad0 con 0x7f21cc0138f0 2021-03-09T10:16:11.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:11.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:11.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:11.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:11.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:11.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:11.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:11.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:11.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:11.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:16:11.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.429+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0552b0 con 0x7f21cc0138f0 2021-03-09T10:16:11.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084ca0 secure :-1 s=READY pgs=10416 cs=0 l=1 rev1=1 rx=0x7f21e4053f80 tx=0x7f21e401ff40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:11.438 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:11.433+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35510 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:11.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024410 con 0x7f21cc0138f0 2021-03-09T10:16:11.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024410 con 0x7f21cc0138f0 2021-03-09T10:16:11.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4005230 con 0x7f21cc0138f0 2021-03-09T10:16:11.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.441+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:11.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.441+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:11.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:11.441+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:16.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:16.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:16.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084ca0 secure :-1 s=READY pgs=10416 cs=0 l=1 rev1=1 rx=0x7f21e4053f80 tx=0x7f21e401ff40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:16.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084ca0 secure :-1 s=READY pgs=10416 cs=0 l=1 rev1=1 rx=0x7f21e4053f80 tx=0x7f21e401ff40).stop 2021-03-09T10:16:16.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:16.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084ca0 unknown :-1 s=CLOSED pgs=10416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:16.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:16.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:16.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:16.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:16:16.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ad0 con 0x7f21c4003cb0 2021-03-09T10:16:16.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:16:16.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:16.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:16.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:16.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:16.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:16.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:16.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:16.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:16.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:16.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d240 con 0x7f21cc042f20 2021-03-09T10:16:16.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.437+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:16:16.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.441+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046180 secure :-1 s=READY pgs=10427 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e408f230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:16.445 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:16.441+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35547 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:16.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T10:16:16.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4091120 con 0x7f21cc042f20 2021-03-09T10:16:16.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc042f20 2021-03-09T10:16:16.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.449+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:16.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.449+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:16.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:16.449+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:21.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:21.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:21.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046180 secure :-1 s=READY pgs=10427 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e408f230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:21.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046180 secure :-1 s=READY pgs=10427 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e408f230).stop 2021-03-09T10:16:21.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:21.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046180 unknown :-1 s=CLOSED pgs=10427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:21.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:21.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:21.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:21.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ad0 con 0x7f21cc005e30 2021-03-09T10:16:21.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:16:21.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03fd80 con 0x7f21cc0138f0 2021-03-09T10:16:21.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:21.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:21.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:21.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:21.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:21.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:21.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:21.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:21.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:21.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:16:21.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc0138f0 2021-03-09T10:16:21.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.445+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10435 cs=0 l=1 rev1=1 rx=0x7f21e0098f30 tx=0x7f21e000b490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:21.452 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:21.449+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35585 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:21.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:16:21.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:16:21.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:16:21.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.457+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:21.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.461+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:21.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:21.461+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:25.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:25.367 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:26.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:26.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:26.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10435 cs=0 l=1 rev1=1 rx=0x7f21e0098f30 tx=0x7f21e000b490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:26.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10435 cs=0 l=1 rev1=1 rx=0x7f21e0098f30 tx=0x7f21e000b490).stop 2021-03-09T10:16:26.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:26.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=10435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:26.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:26.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:26.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:26.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ad0 con 0x7f21cc005e30 2021-03-09T10:16:26.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465d0 con 0x7f21c4003cb0 2021-03-09T10:16:26.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc042f20 2021-03-09T10:16:26.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:26.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:26.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:26.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:26.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.449+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:26.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:26.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:26.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:26.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:26.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017e70 con 0x7f21cc042f20 2021-03-09T10:16:26.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d240 con 0x7f21cc042f20 2021-03-09T10:16:26.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10450 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e40244e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:26.462 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:26.457+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35622 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:26.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1e90 con 0x7f21cc042f20 2021-03-09T10:16:26.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408a990 con 0x7f21cc042f20 2021-03-09T10:16:26.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc042f20 2021-03-09T10:16:26.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.465+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:26.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:26.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:26.465+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:31.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:31.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:31.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10450 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e40244e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:31.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10450 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e40244e0).stop 2021-03-09T10:16:31.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:31.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:31.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:31.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:31.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:31.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ad0 con 0x7f21cc005e30 2021-03-09T10:16:31.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:16:31.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d240 con 0x7f21cc0138f0 2021-03-09T10:16:31.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:31.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:31.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:31.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:31.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:31.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:31.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:31.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:31.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:31.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0263e0 con 0x7f21cc0138f0 2021-03-09T10:16:31.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:16:31.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10456 cs=0 l=1 rev1=1 rx=0x7f21e0007810 tx=0x7f21e0075c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:31.468 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:31.461+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35660 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:31.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21cc0138f0 2021-03-09T10:16:31.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21cc0138f0 2021-03-09T10:16:31.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:16:31.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.469+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:31.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:31.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:31.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:36.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:36.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:36.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10456 cs=0 l=1 rev1=1 rx=0x7f21e0007810 tx=0x7f21e0075c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:36.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10456 cs=0 l=1 rev1=1 rx=0x7f21e0007810 tx=0x7f21e0075c20).stop 2021-03-09T10:16:36.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:36.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=10456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:36.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:36.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:36.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:36.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:36.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089830 con 0x7f21cc005e30 2021-03-09T10:16:36.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:36.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ad0 con 0x7f21c4003cb0 2021-03-09T10:16:36.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc042f20 2021-03-09T10:16:36.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:36.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:36.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:36.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:36.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:36.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:36.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:36.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d240 con 0x7f21cc042f20 2021-03-09T10:16:36.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017e70 con 0x7f21cc042f20 2021-03-09T10:16:36.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.469+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10468 cs=0 l=1 rev1=1 rx=0x7f21cc084a60 tx=0x7f21e4091120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:36.475 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:36.469+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35697 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:36.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc042f20 2021-03-09T10:16:36.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:16:36.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc042f20 2021-03-09T10:16:36.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.477+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:36.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:36.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:36.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:41.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:41.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:41.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10468 cs=0 l=1 rev1=1 rx=0x7f21cc084a60 tx=0x7f21e4091120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:41.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10468 cs=0 l=1 rev1=1 rx=0x7f21cc084a60 tx=0x7f21e4091120).stop 2021-03-09T10:16:41.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:41.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:41.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:41.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:41.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:41.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:16:41.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041390 con 0x7f21c4003cb0 2021-03-09T10:16:41.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0897f0 con 0x7f21cc0138f0 2021-03-09T10:16:41.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:41.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:41.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:41.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:41.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:41.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:41.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:41.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:41.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:41.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:16:41.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:16:41.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.473+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10478 cs=0 l=1 rev1=1 rx=0x7f21e007c6f0 tx=0x7f21e0013c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:41.482 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:41.477+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35735 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:41.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0013e50 con 0x7f21cc0138f0 2021-03-09T10:16:41.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0013e50 con 0x7f21cc0138f0 2021-03-09T10:16:41.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:16:41.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.485+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:41.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.485+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:41.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:41.485+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:46.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:46.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:46.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10478 cs=0 l=1 rev1=1 rx=0x7f21e007c6f0 tx=0x7f21e0013c80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:46.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10478 cs=0 l=1 rev1=1 rx=0x7f21e007c6f0 tx=0x7f21e0013c80).stop 2021-03-09T10:16:46.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:46.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=10478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:46.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:46.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:46.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:46.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:16:46.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007f40 con 0x7f21c4003cb0 2021-03-09T10:16:46.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465d0 con 0x7f21cc042f20 2021-03-09T10:16:46.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:46.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:46.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:46.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:46.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:46.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:46.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:46.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:46.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:46.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0897f0 con 0x7f21cc042f20 2021-03-09T10:16:46.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc042f20 2021-03-09T10:16:46.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.481+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10488 cs=0 l=1 rev1=1 rx=0x7f21e0076f40 tx=0x7f21e007cbd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:46.489 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:46.485+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35772 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:46.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21cc042f20 2021-03-09T10:16:46.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21cc042f20 2021-03-09T10:16:46.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.489+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:16:46.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.489+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:46.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.489+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:46.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:46.489+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:51.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:51.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:51.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10488 cs=0 l=1 rev1=1 rx=0x7f21e0076f40 tx=0x7f21e007cbd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:51.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10488 cs=0 l=1 rev1=1 rx=0x7f21e0076f40 tx=0x7f21e007cbd0).stop 2021-03-09T10:16:51.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:51.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:51.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:51.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:51.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:51.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027e80 con 0x7f21cc005e30 2021-03-09T10:16:51.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21c4003cb0 2021-03-09T10:16:51.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:16:51.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:51.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:51.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:51.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:51.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.485+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:51.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:51.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:51.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:51.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:51.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007ef0 con 0x7f21cc0138f0 2021-03-09T10:16:51.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d240 con 0x7f21cc0138f0 2021-03-09T10:16:51.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.489+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10501 cs=0 l=1 rev1=1 rx=0x7f21cc0955e0 tx=0x7f21e40b11d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:51.496 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:51.493+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35810 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:51.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.493+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400f310 con 0x7f21cc0138f0 2021-03-09T10:16:51.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.497+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400f310 con 0x7f21cc0138f0 2021-03-09T10:16:51.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.497+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:16:51.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.497+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:51.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.497+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:51.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:51.497+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:16:55.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:55.438 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:16:56.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:56.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:56.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10501 cs=0 l=1 rev1=1 rx=0x7f21cc0955e0 tx=0x7f21e40b11d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:56.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10501 cs=0 l=1 rev1=1 rx=0x7f21cc0955e0 tx=0x7f21e40b11d0).stop 2021-03-09T10:16:56.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:56.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=10501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:56.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:56.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:56.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:56.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027e80 con 0x7f21cc005e30 2021-03-09T10:16:56.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dcb0 con 0x7f21c4003cb0 2021-03-09T10:16:56.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc042f20 2021-03-09T10:16:56.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:16:56.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:56.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:16:56.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:56.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:56.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:56.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:56.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:56.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:56.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d240 con 0x7f21cc042f20 2021-03-09T10:16:56.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096d10 con 0x7f21cc042f20 2021-03-09T10:16:56.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10512 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:56.504 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:16:56.496+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35847 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:16:56.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:16:56.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:16:56.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:16:56.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.508+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:16:56.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:16:56.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:16:56.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:01.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:01.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:01.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10512 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:01.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10512 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc009290).stop 2021-03-09T10:17:01.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:01.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:01.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:01.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:01.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:01.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc083ed0 con 0x7f21cc005e30 2021-03-09T10:17:01.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dc80 con 0x7f21c4003cb0 2021-03-09T10:17:01.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc050070 con 0x7f21cc0138f0 2021-03-09T10:17:01.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:01.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:01.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:01.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:01.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.500+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:01.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:01.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:01.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:01.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:01.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:17:01.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0897f0 con 0x7f21cc0138f0 2021-03-09T10:17:01.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10521 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0013cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:01.513 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:01.504+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35885 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:01.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc0138f0 2021-03-09T10:17:01.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:17:01.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:17:01.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.512+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:01.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.512+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:01.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:01.512+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:06.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:06.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:06.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10521 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0013cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:06.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10521 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0013cf0).stop 2021-03-09T10:17:06.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:06.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=10521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:06.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:06.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:06.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:06.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:17:06.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465d0 con 0x7f21c4003cb0 2021-03-09T10:17:06.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012710 con 0x7f21cc042f20 2021-03-09T10:17:06.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:06.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:06.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:06.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:06.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:06.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:06.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:06.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:06.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:06.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012850 con 0x7f21cc042f20 2021-03-09T10:17:06.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.508+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc042f20 2021-03-09T10:17:06.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10531 cs=0 l=1 rev1=1 rx=0x7f21e0075d50 tx=0x7f21e003c000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:06.519 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:06.516+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35922 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:06.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0040 con 0x7f21cc042f20 2021-03-09T10:17:06.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc042f20 2021-03-09T10:17:06.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:17:06.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.516+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:06.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.520+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:06.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:06.520+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:11.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:11.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:11.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10531 cs=0 l=1 rev1=1 rx=0x7f21e0075d50 tx=0x7f21e003c000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:11.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10531 cs=0 l=1 rev1=1 rx=0x7f21e0075d50 tx=0x7f21e003c000).stop 2021-03-09T10:17:11.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:11.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:11.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:11.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:11.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:11.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:17:11.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012710 con 0x7f21c4003cb0 2021-03-09T10:17:11.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012850 con 0x7f21cc0138f0 2021-03-09T10:17:11.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:11.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:11.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:11.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:11.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.516+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:11.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:11.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:11.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:11.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:11.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026350 con 0x7f21cc0138f0 2021-03-09T10:17:11.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d240 con 0x7f21cc0138f0 2021-03-09T10:17:11.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.520+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10545 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc081090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:11.528 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:11.524+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35960 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:11.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0691f0 con 0x7f21cc0138f0 2021-03-09T10:17:11.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.528+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0691f0 con 0x7f21cc0138f0 2021-03-09T10:17:11.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:17:11.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.532+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:11.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.532+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:11.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:11.532+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:16.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:17:16.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:16.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10545 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc081090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:16.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=10545 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc081090).stop 2021-03-09T10:17:16.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:16.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=10545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:16.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:16.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:16.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:16.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d290 con 0x7f21cc005e30 2021-03-09T10:17:16.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:17:16.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465d0 con 0x7f21cc042f20 2021-03-09T10:17:16.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:16.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:16.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:16.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:16.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:16.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:16.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:16.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:16.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:16.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012570 con 0x7f21cc042f20 2021-03-09T10:17:16.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc042f20 2021-03-09T10:17:16.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10552 cs=0 l=1 rev1=1 rx=0x7f21e0006680 tx=0x7f21e002bf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:16.535 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:16.532+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35997 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:16.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:17:16.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003dbd0 con 0x7f21cc042f20 2021-03-09T10:17:16.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.536+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:17:16.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.536+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:16.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.536+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:16.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:16.536+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:21.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:21.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:21.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10552 cs=0 l=1 rev1=1 rx=0x7f21e0006680 tx=0x7f21e002bf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:21.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10552 cs=0 l=1 rev1=1 rx=0x7f21e0006680 tx=0x7f21e002bf60).stop 2021-03-09T10:17:21.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:21.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:21.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:21.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:21.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:21.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cfc0 con 0x7f21cc005e30 2021-03-09T10:17:21.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012700 con 0x7f21c4003cb0 2021-03-09T10:17:21.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012840 con 0x7f21cc0138f0 2021-03-09T10:17:21.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:21.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:21.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:21.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:21.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.532+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:21.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:21.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:21.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:21.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:21.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0129b0 con 0x7f21cc0138f0 2021-03-09T10:17:21.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007ef0 con 0x7f21cc0138f0 2021-03-09T10:17:21.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012c80 secure :-1 s=READY pgs=10565 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e400f320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:21.543 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:21.536+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36035 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:21.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:17:21.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009e30 con 0x7f21cc0138f0 2021-03-09T10:17:21.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:17:21.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.544+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:21.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.544+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:21.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:21.544+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:25.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:25.479 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:26.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:26.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:26.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012c80 secure :-1 s=READY pgs=10565 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e400f320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:26.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012c80 secure :-1 s=READY pgs=10565 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e400f320).stop 2021-03-09T10:17:26.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:26.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012c80 unknown :-1 s=CLOSED pgs=10565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:26.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:26.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:26.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:26.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cfc0 con 0x7f21cc005e30 2021-03-09T10:17:26.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040a60 con 0x7f21c4003cb0 2021-03-09T10:17:26.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007d90 con 0x7f21cc042f20 2021-03-09T10:17:26.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:26.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:26.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:26.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:26.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:26.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:26.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:26.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:26.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:26.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc007ef0 con 0x7f21cc042f20 2021-03-09T10:17:26.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026350 con 0x7f21cc042f20 2021-03-09T10:17:26.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10574 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc015a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:26.550 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:26.544+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36072 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:26.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.548+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089870 con 0x7f21cc042f20 2021-03-09T10:17:26.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.552+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089870 con 0x7f21cc042f20 2021-03-09T10:17:26.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.552+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc042f20 2021-03-09T10:17:26.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.552+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:26.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.552+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:26.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:26.552+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:31.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:31.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:31.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10574 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc015a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:31.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10574 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc015a20).stop 2021-03-09T10:17:31.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:31.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:31.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:31.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:31.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:31.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cfc0 con 0x7f21cc005e30 2021-03-09T10:17:31.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007d90 con 0x7f21c4003cb0 2021-03-09T10:17:31.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040a40 con 0x7f21cc0138f0 2021-03-09T10:17:31.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:31.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:31.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:31.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:31.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:31.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:31.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:31.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:31.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:31.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026350 con 0x7f21cc0138f0 2021-03-09T10:17:31.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012570 con 0x7f21cc0138f0 2021-03-09T10:17:31.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.548+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10584 cs=0 l=1 rev1=1 rx=0x7f21e002c040 tx=0x7f21e003c070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:31.557 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:31.552+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36110 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:31.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.552+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc0138f0 2021-03-09T10:17:31.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:17:31.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:17:31.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.560+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:31.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.564+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:31.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:31.564+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:36.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:36.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:36.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10584 cs=0 l=1 rev1=1 rx=0x7f21e002c040 tx=0x7f21e003c070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:36.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10584 cs=0 l=1 rev1=1 rx=0x7f21e002c040 tx=0x7f21e003c070).stop 2021-03-09T10:17:36.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:36.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:36.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:36.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:36.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:36.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cfc0 con 0x7f21cc005e30 2021-03-09T10:17:36.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040a40 con 0x7f21c4003cb0 2021-03-09T10:17:36.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc045440 con 0x7f21cc042f20 2021-03-09T10:17:36.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:36.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:36.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:36.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:36.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:36.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:36.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:36.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:36.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:36.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0264f0 con 0x7f21cc042f20 2021-03-09T10:17:36.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc007ef0 con 0x7f21cc042f20 2021-03-09T10:17:36.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10596 cs=0 l=1 rev1=1 rx=0x7f21dc081060 tx=0x7f21dc052bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:36.566 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:36.560+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36147 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:36.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc042f20 2021-03-09T10:17:36.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3280 con 0x7f21cc042f20 2021-03-09T10:17:36.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc042f20 2021-03-09T10:17:36.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.564+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:36.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.564+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:36.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:36.564+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:41.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:41.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:41.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10596 cs=0 l=1 rev1=1 rx=0x7f21dc081060 tx=0x7f21dc052bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:41.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10596 cs=0 l=1 rev1=1 rx=0x7f21dc081060 tx=0x7f21dc052bc0).stop 2021-03-09T10:17:41.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:41.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:41.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:41.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:41.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:41.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d470 con 0x7f21cc005e30 2021-03-09T10:17:41.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040a40 con 0x7f21c4003cb0 2021-03-09T10:17:41.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007ef0 con 0x7f21cc0138f0 2021-03-09T10:17:41.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:41.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:41.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:41.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:41.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.560+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:41.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:41.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:41.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:41.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:41.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026350 con 0x7f21cc0138f0 2021-03-09T10:17:41.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0129b0 con 0x7f21cc0138f0 2021-03-09T10:17:41.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.564+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10607 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e408f2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:41.572 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:41.568+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36185 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:41.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024210 con 0x7f21cc0138f0 2021-03-09T10:17:41.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024210 con 0x7f21cc0138f0 2021-03-09T10:17:41.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:17:41.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.572+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:41.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.572+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:41.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:41.572+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:46.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:46.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:46.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10607 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e408f2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:46.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10607 cs=0 l=1 rev1=1 rx=0x7f21e408f1c0 tx=0x7f21e408f2d0).stop 2021-03-09T10:17:46.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:46.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:46.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:46.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:46.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:46.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026350 con 0x7f21cc005e30 2021-03-09T10:17:46.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465d0 con 0x7f21c4003cb0 2021-03-09T10:17:46.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035870 con 0x7f21cc042f20 2021-03-09T10:17:46.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:46.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:46.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:46.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:46.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:46.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:46.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:46.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:46.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:46.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0359b0 con 0x7f21cc042f20 2021-03-09T10:17:46.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0129b0 con 0x7f21cc042f20 2021-03-09T10:17:46.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10618 cs=0 l=1 rev1=1 rx=0x7f21cc021f70 tx=0x7f21e4091120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:46.579 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:46.571+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36222 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:46.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400de70 con 0x7f21cc042f20 2021-03-09T10:17:46.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400de70 con 0x7f21cc042f20 2021-03-09T10:17:46.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a5d0 con 0x7f21cc042f20 2021-03-09T10:17:46.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.579+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:46.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.579+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:46.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:46.579+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:51.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:17:51.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:51.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10618 cs=0 l=1 rev1=1 rx=0x7f21cc021f70 tx=0x7f21e4091120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:51.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10618 cs=0 l=1 rev1=1 rx=0x7f21cc021f70 tx=0x7f21e4091120).stop 2021-03-09T10:17:51.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:51.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:51.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:51.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:51.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:51.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026350 con 0x7f21cc005e30 2021-03-09T10:17:51.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035720 con 0x7f21c4003cb0 2021-03-09T10:17:51.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035860 con 0x7f21cc0138f0 2021-03-09T10:17:51.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:51.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:51.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:51.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:51.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:51.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:51.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:51.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:51.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:51.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc097090 con 0x7f21cc0138f0 2021-03-09T10:17:51.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:17:51.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10628 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:51.587 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:51.579+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36260 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:51.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:17:51.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:17:51.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:17:51.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.583+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:51.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.587+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:51.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:51.587+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:17:55.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:55.527 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:17:56.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:56.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:56.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10628 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:56.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10628 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d620).stop 2021-03-09T10:17:56.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:56.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:56.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:56.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:56.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:56.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097220 con 0x7f21cc005e30 2021-03-09T10:17:56.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465d0 con 0x7f21c4003cb0 2021-03-09T10:17:56.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040de0 con 0x7f21cc042f20 2021-03-09T10:17:56.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:17:56.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:56.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:17:56.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:56.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:56.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:56.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:56.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:56.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:56.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:17:56.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0264f0 con 0x7f21cc042f20 2021-03-09T10:17:56.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.587+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10640 cs=0 l=1 rev1=1 rx=0x7f21cc0495a0 tx=0x7f21dc010e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:56.594 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:17:56.591+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36296 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:17:56.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0695d0 con 0x7f21cc042f20 2021-03-09T10:17:56.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0695d0 con 0x7f21cc042f20 2021-03-09T10:17:56.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b39f0 con 0x7f21cc042f20 2021-03-09T10:17:56.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.595+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:17:56.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:17:56.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:17:56.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:01.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:01.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:01.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10640 cs=0 l=1 rev1=1 rx=0x7f21cc0495a0 tx=0x7f21dc010e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:01.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 secure :-1 s=READY pgs=10640 cs=0 l=1 rev1=1 rx=0x7f21cc0495a0 tx=0x7f21dc010e10).stop 2021-03-09T10:18:01.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:01.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460e0 unknown :-1 s=CLOSED pgs=10640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:01.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:01.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:01.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:01.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097220 con 0x7f21cc005e30 2021-03-09T10:18:01.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040db0 con 0x7f21c4003cb0 2021-03-09T10:18:01.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040ef0 con 0x7f21cc0138f0 2021-03-09T10:18:01.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:01.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:01.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:01.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:01.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:01.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:01.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:01.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:01.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:01.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026350 con 0x7f21cc0138f0 2021-03-09T10:18:01.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0359b0 con 0x7f21cc0138f0 2021-03-09T10:18:01.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10648 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4017190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:01.603 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:01.599+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36335 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:01.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5ec0 con 0x7f21cc0138f0 2021-03-09T10:18:01.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5ec0 con 0x7f21cc0138f0 2021-03-09T10:18:01.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc0138f0 2021-03-09T10:18:01.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.611+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:01.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.611+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:01.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:01.611+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:06.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:06.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:06.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10648 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4017190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:06.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10648 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4017190).stop 2021-03-09T10:18:06.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:06.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:06.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:06.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:06.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:06.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097220 con 0x7f21cc005e30 2021-03-09T10:18:06.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0359b0 con 0x7f21c4003cb0 2021-03-09T10:18:06.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465a0 con 0x7f21cc042f20 2021-03-09T10:18:06.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:06.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:06.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:06.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:06.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.599+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:06.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:06.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:06.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:06.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:06.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040d60 con 0x7f21cc042f20 2021-03-09T10:18:06.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc097090 con 0x7f21cc042f20 2021-03-09T10:18:06.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.603+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460b0 secure :-1 s=READY pgs=10660 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007cb70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:06.608 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:06.607+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36371 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:06.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:18:06.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0098c00 con 0x7f21cc042f20 2021-03-09T10:18:06.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:18:06.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.611+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:06.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.611+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:06.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:06.611+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:11.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:11.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:11.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460b0 secure :-1 s=READY pgs=10660 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007cb70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:11.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460b0 secure :-1 s=READY pgs=10660 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007cb70).stop 2021-03-09T10:18:11.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0460b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:11.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0460b0 unknown :-1 s=CLOSED pgs=10660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:11.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:11.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:11.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:11.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057b90 con 0x7f21cc005e30 2021-03-09T10:18:11.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042870 con 0x7f21c4003cb0 2021-03-09T10:18:11.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097060 con 0x7f21cc0138f0 2021-03-09T10:18:11.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:11.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:11.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:11.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:11.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:11.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:11.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:11.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:11.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:11.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0971a0 con 0x7f21cc0138f0 2021-03-09T10:18:11.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:18:11.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10670 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc080040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:11.616 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:11.615+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36410 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:11.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083ab0 con 0x7f21cc0138f0 2021-03-09T10:18:11.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083ab0 con 0x7f21cc0138f0 2021-03-09T10:18:11.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:18:11.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.619+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:11.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.623+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:11.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:11.623+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:16.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:16.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:16.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10670 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc080040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:16.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10670 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc080040).stop 2021-03-09T10:18:16.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:16.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:16.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:16.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:16.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:16.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046570 con 0x7f21cc005e30 2021-03-09T10:18:16.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036b70 con 0x7f21c4003cb0 2021-03-09T10:18:16.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036cb0 con 0x7f21cc042f20 2021-03-09T10:18:16.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:16.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:16.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:16.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:16.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.615+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:16.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:16.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:16.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:16.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:16.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:18:16.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026350 con 0x7f21cc042f20 2021-03-09T10:18:16.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.619+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046080 secure :-1 s=READY pgs=10681 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e400f2b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:16.628 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:16.623+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36446 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:16.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21cc042f20 2021-03-09T10:18:16.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21cc042f20 2021-03-09T10:18:16.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:18:16.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.631+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:16.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.631+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:16.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:16.631+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:21.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:21.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:21.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046080 secure :-1 s=READY pgs=10681 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e400f2b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:21.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046080 secure :-1 s=READY pgs=10681 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e400f2b0).stop 2021-03-09T10:18:21.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:21.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046080 unknown :-1 s=CLOSED pgs=10681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:21.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:21.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:21.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:21.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0264f0 con 0x7f21cc005e30 2021-03-09T10:18:21.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0369d0 con 0x7f21c4003cb0 2021-03-09T10:18:21.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036b10 con 0x7f21cc0138f0 2021-03-09T10:18:21.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:21.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:21.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:21.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:21.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:21.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:21.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:21.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:21.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:21.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036c50 con 0x7f21cc0138f0 2021-03-09T10:18:21.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:18:21.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.627+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10691 cs=0 l=1 rev1=1 rx=0x7f21cc035d70 tx=0x7f21e4017190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:21.632 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:21.627+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36485 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:21.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024210 con 0x7f21cc0138f0 2021-03-09T10:18:21.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21cc0138f0 2021-03-09T10:18:21.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.635+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21cc0138f0 2021-03-09T10:18:21.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.635+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:21.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.635+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:21.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:21.635+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:25.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:25.574 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:26.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:26.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:26.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10691 cs=0 l=1 rev1=1 rx=0x7f21cc035d70 tx=0x7f21e4017190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:26.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10691 cs=0 l=1 rev1=1 rx=0x7f21cc035d70 tx=0x7f21e4017190).stop 2021-03-09T10:18:26.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:26.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:26.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:26.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:26.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:26.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d880 con 0x7f21cc005e30 2021-03-09T10:18:26.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0264f0 con 0x7f21c4003cb0 2021-03-09T10:18:26.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0464b0 con 0x7f21cc042f20 2021-03-09T10:18:26.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:26.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:26.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:26.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:26.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.631+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:26.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:26.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:26.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:26.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:26.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:18:26.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040d60 con 0x7f21cc042f20 2021-03-09T10:18:26.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045fc0 secure :-1 s=READY pgs=10705 cs=0 l=1 rev1=1 rx=0x7f21e004dfa0 tx=0x7f21e007e4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:26.638 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:26.635+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36521 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:26.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21cc042f20 2021-03-09T10:18:26.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.643+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:18:26.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.643+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:18:26.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.643+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:26.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.643+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:26.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:26.643+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:31.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:31.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:31.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045fc0 secure :-1 s=READY pgs=10705 cs=0 l=1 rev1=1 rx=0x7f21e004dfa0 tx=0x7f21e007e4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:31.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045fc0 secure :-1 s=READY pgs=10705 cs=0 l=1 rev1=1 rx=0x7f21e004dfa0 tx=0x7f21e007e4e0).stop 2021-03-09T10:18:31.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:31.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045fc0 unknown :-1 s=CLOSED pgs=10705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:31.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.635+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:31.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:31.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:31.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026120 con 0x7f21cc005e30 2021-03-09T10:18:31.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040d10 con 0x7f21c4003cb0 2021-03-09T10:18:31.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040e50 con 0x7f21cc0138f0 2021-03-09T10:18:31.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:31.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:31.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:31.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:31.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:31.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:31.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:31.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:31.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:31.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:18:31.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0971a0 con 0x7f21cc0138f0 2021-03-09T10:18:31.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.639+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10712 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0098f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:31.646 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:31.643+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36560 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:31.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020340 con 0x7f21cc0138f0 2021-03-09T10:18:31.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:18:31.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020340 con 0x7f21cc0138f0 2021-03-09T10:18:31.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.651+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:31.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.651+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:31.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:31.651+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:36.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.642+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:36.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.642+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:36.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.642+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10712 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0098f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:36.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.642+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10712 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0098f0).stop 2021-03-09T10:18:36.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:36.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:36.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:36.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:36.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:36.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0309f0 con 0x7f21cc005e30 2021-03-09T10:18:36.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:36.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:36.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:36.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046420 con 0x7f21c4003cb0 2021-03-09T10:18:36.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:36.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046560 con 0x7f21cc042f20 2021-03-09T10:18:36.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:36.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:36.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:36.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:36.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:36.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc040cc0 con 0x7f21cc042f20 2021-03-09T10:18:36.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.646+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036c50 con 0x7f21cc042f20 2021-03-09T10:18:36.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.650+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10723 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e4091240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:36.655 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:36.650+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36596 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:36.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400f320 con 0x7f21cc042f20 2021-03-09T10:18:36.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400f320 con 0x7f21cc042f20 2021-03-09T10:18:36.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T10:18:36.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.658+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:36.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.658+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:36.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:36.658+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:41.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:41.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:41.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10723 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e4091240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:41.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10723 cs=0 l=1 rev1=1 rx=0x7f21e40912b0 tx=0x7f21e4091240).stop 2021-03-09T10:18:41.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:41.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=CLOSED pgs=10723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:41.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:41.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:41.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:41.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015f80 con 0x7f21cc005e30 2021-03-09T10:18:41.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0309f0 con 0x7f21c4003cb0 2021-03-09T10:18:41.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21cc0138f0 2021-03-09T10:18:41.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:41.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:41.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:41.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.650+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:41.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:41.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:41.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:41.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:41.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:41.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036df0 con 0x7f21cc0138f0 2021-03-09T10:18:41.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:18:41.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.654+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10735 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e0004960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:41.662 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:41.658+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36635 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:41.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21cc0138f0 2021-03-09T10:18:41.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0090080 con 0x7f21cc0138f0 2021-03-09T10:18:41.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.662+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:18:41.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.662+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:41.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.666+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:41.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:41.666+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:46.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:46.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:46.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10735 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e0004960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:46.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10735 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e0004960).stop 2021-03-09T10:18:46.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:46.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:46.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:46.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:46.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:46.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015f60 con 0x7f21cc005e30 2021-03-09T10:18:46.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046420 con 0x7f21c4003cb0 2021-03-09T10:18:46.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:46.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046560 con 0x7f21cc042f20 2021-03-09T10:18:46.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:46.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:46.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:46.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.658+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:46.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:46.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:46.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:46.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:46.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084c80 con 0x7f21cc042f20 2021-03-09T10:18:46.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc040cc0 con 0x7f21cc042f20 2021-03-09T10:18:46.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.662+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10744 cs=0 l=1 rev1=1 rx=0x7f21cc012b60 tx=0x7f21e400f3d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:46.670 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:46.666+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36671 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:46.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:18:46.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:18:46.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.670+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:18:46.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.670+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:46.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.670+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:46.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:46.670+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:51.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:51.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:51.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10744 cs=0 l=1 rev1=1 rx=0x7f21cc012b60 tx=0x7f21e400f3d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:51.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10744 cs=0 l=1 rev1=1 rx=0x7f21cc012b60 tx=0x7f21e400f3d0).stop 2021-03-09T10:18:51.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:51.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=CLOSED pgs=10744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:51.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:51.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:51.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:51.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0130d0 con 0x7f21cc005e30 2021-03-09T10:18:51.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c960 con 0x7f21c4003cb0 2021-03-09T10:18:51.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040e60 con 0x7f21cc0138f0 2021-03-09T10:18:51.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:51.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:51.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:51.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.666+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:51.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:51.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:51.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:51.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:51.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:51.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025fd0 con 0x7f21cc0138f0 2021-03-09T10:18:51.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084af0 con 0x7f21cc0138f0 2021-03-09T10:18:51.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10759 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:51.677 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:51.674+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36710 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:51.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:18:51.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.678+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f110 con 0x7f21cc0138f0 2021-03-09T10:18:51.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0138f0 2021-03-09T10:18:51.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.682+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:51.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.682+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:51.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:51.682+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:18:55.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:55.639 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:18:56.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:56.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:56.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10759 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:56.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 secure :-1 s=READY pgs=10759 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc077e50).stop 2021-03-09T10:18:56.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:56.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012be0 unknown :-1 s=CLOSED pgs=10759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:56.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:56.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:56.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:56.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013270 con 0x7f21cc005e30 2021-03-09T10:18:56.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:18:56.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040e60 con 0x7f21cc042f20 2021-03-09T10:18:56.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:18:56.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:18:56.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:56.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:56.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:56.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:56.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:56.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:56.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:56.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046420 con 0x7f21cc042f20 2021-03-09T10:18:56.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.678+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036df0 con 0x7f21cc042f20 2021-03-09T10:18:56.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.682+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10771 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:56.686 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:18:56.682+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36746 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:18:56.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e001fab0 con 0x7f21cc042f20 2021-03-09T10:18:56.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.686+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0014eb0 con 0x7f21cc042f20 2021-03-09T10:18:56.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:18:56.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.690+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:18:56.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.690+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:18:56.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:18:56.690+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:01.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:01.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:01.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10771 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:01.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10771 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003820).stop 2021-03-09T10:19:01.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:01.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=CLOSED pgs=10771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:01.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:01.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:01.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:01.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c30 con 0x7f21cc005e30 2021-03-09T10:19:01.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:19:01.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040e60 con 0x7f21cc0138f0 2021-03-09T10:19:01.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:01.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:01.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:01.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:01.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.682+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:01.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:01.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:01.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:01.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:01.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046420 con 0x7f21cc0138f0 2021-03-09T10:19:01.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025fd0 con 0x7f21cc0138f0 2021-03-09T10:19:01.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.686+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 secure :-1 s=READY pgs=10777 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc062120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:01.693 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:01.690+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36785 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:01.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:19:01.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.694+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:19:01.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.694+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06dcd0 con 0x7f21cc0138f0 2021-03-09T10:19:01.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.694+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:01.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.694+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:01.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:01.694+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:06.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:06.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:06.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 secure :-1 s=READY pgs=10777 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc062120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:06.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 secure :-1 s=READY pgs=10777 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc062120).stop 2021-03-09T10:19:06.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:06.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 unknown :-1 s=CLOSED pgs=10777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:06.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:06.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:06.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:06.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc005e30 2021-03-09T10:19:06.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040e60 con 0x7f21c4003cb0 2021-03-09T10:19:06.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007c30 con 0x7f21cc042f20 2021-03-09T10:19:06.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:06.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:06.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:06.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:06.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.690+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:06.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:06.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:06.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:06.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:06.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046420 con 0x7f21cc042f20 2021-03-09T10:19:06.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21cc042f20 2021-03-09T10:19:06.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.694+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10787 cs=0 l=1 rev1=1 rx=0x7f21e0013a60 tx=0x7f21e00138a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:06.701 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:06.698+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36821 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:06.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:19:06.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:19:06.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:19:06.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.702+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:06.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.702+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:06.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:06.702+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:11.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:11.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:11.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10787 cs=0 l=1 rev1=1 rx=0x7f21e0013a60 tx=0x7f21e00138a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:11.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 secure :-1 s=READY pgs=10787 cs=0 l=1 rev1=1 rx=0x7f21e0013a60 tx=0x7f21e00138a0).stop 2021-03-09T10:19:11.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:11.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f30 unknown :-1 s=CLOSED pgs=10787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:11.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:11.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:11.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:11.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc005e30 2021-03-09T10:19:11.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040e60 con 0x7f21c4003cb0 2021-03-09T10:19:11.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046420 con 0x7f21cc0138f0 2021-03-09T10:19:11.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:11.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:11.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:11.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:11.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.698+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:11.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:11.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:11.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:11.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:11.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013080 con 0x7f21cc0138f0 2021-03-09T10:19:11.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21cc0138f0 2021-03-09T10:19:11.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 secure :-1 s=READY pgs=10798 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0004b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:11.710 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:11.706+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36860 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:11.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21cc0138f0 2021-03-09T10:19:11.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0013d00 con 0x7f21cc0138f0 2021-03-09T10:19:11.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00149a0 con 0x7f21cc0138f0 2021-03-09T10:19:11.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.710+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:11.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.710+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:11.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:11.710+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:16.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:16.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:16.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 secure :-1 s=READY pgs=10798 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0004b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:16.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 secure :-1 s=READY pgs=10798 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0004b30).stop 2021-03-09T10:19:16.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:16.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b90 unknown :-1 s=CLOSED pgs=10798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:16.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:16.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:16.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:16.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc005e30 2021-03-09T10:19:16.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0463f0 con 0x7f21c4003cb0 2021-03-09T10:19:16.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013210 con 0x7f21cc042f20 2021-03-09T10:19:16.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:16.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:16.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:16.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:16.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:16.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:16.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:16.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:16.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.706+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:16.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.710+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013350 con 0x7f21cc042f20 2021-03-09T10:19:16.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.710+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013080 con 0x7f21cc042f20 2021-03-09T10:19:16.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.710+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10808 cs=0 l=1 rev1=1 rx=0x7f21e0090240 tx=0x7f21e00a14c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:16.716 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:16.714+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36896 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:16.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0076c20 con 0x7f21cc042f20 2021-03-09T10:19:16.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21cc042f20 2021-03-09T10:19:16.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00149a0 con 0x7f21cc042f20 2021-03-09T10:19:16.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.718+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:16.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.722+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:16.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:16.722+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:21.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:21.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:21.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10808 cs=0 l=1 rev1=1 rx=0x7f21e0090240 tx=0x7f21e00a14c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:21.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10808 cs=0 l=1 rev1=1 rx=0x7f21e0090240 tx=0x7f21e00a14c0).stop 2021-03-09T10:19:21.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:21.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=CLOSED pgs=10808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:21.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:21.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:21.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:21.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21cc005e30 2021-03-09T10:19:21.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013010 con 0x7f21c4003cb0 2021-03-09T10:19:21.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013150 con 0x7f21cc0138f0 2021-03-09T10:19:21.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:21.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:21.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:21.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:21.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:21.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:21.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:21.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:21.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:21.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013290 con 0x7f21cc0138f0 2021-03-09T10:19:21.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025fd0 con 0x7f21cc0138f0 2021-03-09T10:19:21.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.718+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10819 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc081020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:21.724 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:21.718+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36935 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:21.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020450 con 0x7f21cc0138f0 2021-03-09T10:19:21.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020450 con 0x7f21cc0138f0 2021-03-09T10:19:21.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21cc0138f0 2021-03-09T10:19:21.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.730+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:21.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:21.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:21.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:25.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:25.692 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:26.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:26.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:26.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10819 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc081020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:26.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10819 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc081020).stop 2021-03-09T10:19:26.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:26.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:26.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:26.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:26.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:26.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ed0 con 0x7f21cc005e30 2021-03-09T10:19:26.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084af0 con 0x7f21c4003cb0 2021-03-09T10:19:26.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0848d0 con 0x7f21cc042f20 2021-03-09T10:19:26.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:26.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:26.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:26.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:26.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:26.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:26.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:26.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:26.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:26.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc042f20 2021-03-09T10:19:26.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013290 con 0x7f21cc042f20 2021-03-09T10:19:26.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10831 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc06db50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:26.731 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:26.725+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36971 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:26.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069cd0 con 0x7f21cc042f20 2021-03-09T10:19:26.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21cc042f20 2021-03-09T10:19:26.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069cd0 con 0x7f21cc042f20 2021-03-09T10:19:26.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.733+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:26.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.733+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:26.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:26.733+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:31.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:19:31.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:31.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10831 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc06db50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:31.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10831 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc06db50).stop 2021-03-09T10:19:31.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:31.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=CLOSED pgs=10831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:31.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:31.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:31.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:31.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084b20 con 0x7f21cc005e30 2021-03-09T10:19:31.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0463f0 con 0x7f21c4003cb0 2021-03-09T10:19:31.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fd0 con 0x7f21cc0138f0 2021-03-09T10:19:31.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:31.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:31.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:31.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:31.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:31.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:31.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:31.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:31.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:31.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013010 con 0x7f21cc0138f0 2021-03-09T10:19:31.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084c80 con 0x7f21cc0138f0 2021-03-09T10:19:31.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.729+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10840 cs=0 l=1 rev1=1 rx=0x7f21cc02ee70 tx=0x7f21e40171d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:31.744 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:31.733+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37010 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:31.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc0138f0 2021-03-09T10:19:31.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc0138f0 2021-03-09T10:19:31.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.737+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:19:31.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.737+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:31.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.741+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:31.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:31.741+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:36.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:36.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:36.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10840 cs=0 l=1 rev1=1 rx=0x7f21cc02ee70 tx=0x7f21e40171d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:36.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10840 cs=0 l=1 rev1=1 rx=0x7f21cc02ee70 tx=0x7f21e40171d0).stop 2021-03-09T10:19:36.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:36.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:36.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:36.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:36.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:36.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0463f0 con 0x7f21cc005e30 2021-03-09T10:19:36.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017a50 con 0x7f21c4003cb0 2021-03-09T10:19:36.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017b90 con 0x7f21cc042f20 2021-03-09T10:19:36.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.737+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:36.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.737+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:36.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.737+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:36.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.737+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:36.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.737+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:36.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:36.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:36.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:36.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:36.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21cc042f20 2021-03-09T10:19:36.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013290 con 0x7f21cc042f20 2021-03-09T10:19:36.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10855 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc07cc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:36.747 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37046 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:36.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.741+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc042f20 2021-03-09T10:19:36.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:19:36.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc064db0 con 0x7f21cc042f20 2021-03-09T10:19:36.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.745+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:36.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.749+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:36.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:36.749+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:41.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:41.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:41.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10855 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc07cc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:41.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10855 cs=0 l=1 rev1=1 rx=0x7f21dc020040 tx=0x7f21dc07cc90).stop 2021-03-09T10:19:41.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:41.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=CLOSED pgs=10855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:41.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:41.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:41.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:41.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017b90 con 0x7f21cc005e30 2021-03-09T10:19:41.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21c4003cb0 2021-03-09T10:19:41.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fd0 con 0x7f21cc0138f0 2021-03-09T10:19:41.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:41.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:41.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:41.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:41.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:41.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:41.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:41.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:41.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:41.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0131a0 con 0x7f21cc0138f0 2021-03-09T10:19:41.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21cc0138f0 2021-03-09T10:19:41.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10865 cs=0 l=1 rev1=1 rx=0x7f21cc055770 tx=0x7f21dc052070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:41.753 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:41.749+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37085 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:41.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:19:41.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:19:41.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06dc50 con 0x7f21cc0138f0 2021-03-09T10:19:41.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.757+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:41.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.757+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:41.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:41.757+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:46.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:46.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:46.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10865 cs=0 l=1 rev1=1 rx=0x7f21cc055770 tx=0x7f21dc052070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:46.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10865 cs=0 l=1 rev1=1 rx=0x7f21cc055770 tx=0x7f21dc052070).stop 2021-03-09T10:19:46.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:46.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:46.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:46.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:46.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:46.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21cc005e30 2021-03-09T10:19:46.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fd0 con 0x7f21c4003cb0 2021-03-09T10:19:46.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0463f0 con 0x7f21cc042f20 2021-03-09T10:19:46.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:46.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:46.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:46.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:46.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:46.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:46.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:46.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:46.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:46.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084b20 con 0x7f21cc042f20 2021-03-09T10:19:46.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0131a0 con 0x7f21cc042f20 2021-03-09T10:19:46.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10872 cs=0 l=1 rev1=1 rx=0x7f21cc040090 tx=0x7f21dc0ac2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:46.761 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:46.757+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37121 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:46.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0109c0 con 0x7f21cc042f20 2021-03-09T10:19:46.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0109c0 con 0x7f21cc042f20 2021-03-09T10:19:46.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:19:46.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.761+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:46.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.761+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:46.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:46.761+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:51.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:51.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:51.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10872 cs=0 l=1 rev1=1 rx=0x7f21cc040090 tx=0x7f21dc0ac2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:51.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10872 cs=0 l=1 rev1=1 rx=0x7f21cc040090 tx=0x7f21dc0ac2c0).stop 2021-03-09T10:19:51.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:51.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=CLOSED pgs=10872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:51.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:51.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:51.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:51.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21cc005e30 2021-03-09T10:19:51.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fd0 con 0x7f21c4003cb0 2021-03-09T10:19:51.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0131a0 con 0x7f21cc0138f0 2021-03-09T10:19:51.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:51.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:51.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:51.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:51.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:51.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:51.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:51.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:51.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:51.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0132e0 con 0x7f21cc0138f0 2021-03-09T10:19:51.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013010 con 0x7f21cc0138f0 2021-03-09T10:19:51.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10883 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e4016b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:51.768 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:51.765+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37159 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:51.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:19:51.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:19:51.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:19:51.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.769+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:51.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.769+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:51.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:51.769+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:19:55.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:55.741 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:19:56.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:56.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:56.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10883 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e4016b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:56.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 secure :-1 s=READY pgs=10883 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e4016b20).stop 2021-03-09T10:19:56.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:56.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:56.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:56.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:56.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:56.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21cc005e30 2021-03-09T10:19:56.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21c4003cb0 2021-03-09T10:19:56.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dad0 con 0x7f21cc042f20 2021-03-09T10:19:56.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:19:56.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:56.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:19:56.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:56.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:56.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:56.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.765+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:56.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.769+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:56.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.769+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:56.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.769+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc042f20 2021-03-09T10:19:56.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.769+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21cc042f20 2021-03-09T10:19:56.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.769+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10894 cs=0 l=1 rev1=1 rx=0x7f21e0003550 tx=0x7f21e000bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:56.775 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:19:56.773+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37196 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:19:56.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:19:56.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a43a0 con 0x7f21cc042f20 2021-03-09T10:19:56.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:19:56.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.777+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:19:56.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:19:56.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:19:56.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:01.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:20:01.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:01.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10894 cs=0 l=1 rev1=1 rx=0x7f21e0003550 tx=0x7f21e000bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:01.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10894 cs=0 l=1 rev1=1 rx=0x7f21e0003550 tx=0x7f21e000bfd0).stop 2021-03-09T10:20:01.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:01.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=CLOSED pgs=10894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:01.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:01.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0277c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:01.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:01.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21cc005e30 2021-03-09T10:20:01.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21c4003cb0 2021-03-09T10:20:01.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052c30 con 0x7f21cc0138f0 2021-03-09T10:20:01.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:01.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:01.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:01.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:01.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0277c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:01.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:01.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:01.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:01.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:01.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc0138f0 2021-03-09T10:20:01.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.773+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0132e0 con 0x7f21cc0138f0 2021-03-09T10:20:01.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0277c0 secure :-1 s=READY pgs=10904 cs=0 l=1 rev1=1 rx=0x7f21cc032770 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:01.784 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:01.781+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37232 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:01.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21cc0138f0 2021-03-09T10:20:01.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:20:01.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21cc0138f0 2021-03-09T10:20:01.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.785+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:01.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.785+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:01.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:01.785+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:06.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0277c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:06.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0277c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:06.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0277c0 secure :-1 s=READY pgs=10904 cs=0 l=1 rev1=1 rx=0x7f21cc032770 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:06.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0277c0 secure :-1 s=READY pgs=10904 cs=0 l=1 rev1=1 rx=0x7f21cc032770 tx=0x7f21e4053ed0).stop 2021-03-09T10:20:06.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0277c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:06.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0277c0 unknown :-1 s=CLOSED pgs=10904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:06.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:06.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:06.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:06.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21cc005e30 2021-03-09T10:20:06.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21c4003cb0 2021-03-09T10:20:06.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052c30 con 0x7f21cc042f20 2021-03-09T10:20:06.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:06.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:06.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:06.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:06.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:06.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:06.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:06.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:06.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:06.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc042f20 2021-03-09T10:20:06.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0132e0 con 0x7f21cc042f20 2021-03-09T10:20:06.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10922 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e407f1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:06.794 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:06.789+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37271 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:06.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T10:20:06.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:20:06.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T10:20:06.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.793+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:06.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:06.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:06.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:11.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:11.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:11.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10922 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e407f1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:11.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 secure :-1 s=READY pgs=10922 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e407f1e0).stop 2021-03-09T10:20:11.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc045f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:11.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc045f00 unknown :-1 s=CLOSED pgs=10922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:11.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:11.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:11.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc027740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:11.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d0a0 con 0x7f21cc005e30 2021-03-09T10:20:11.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21c4003cb0 2021-03-09T10:20:11.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc052c30 con 0x7f21cc0138f0 2021-03-09T10:20:11.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:11.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:11.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:11.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc027740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:11.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:11.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:11.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:11.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:11.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:11.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc0138f0 2021-03-09T10:20:11.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084b20 con 0x7f21cc0138f0 2021-03-09T10:20:11.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc027740 secure :-1 s=READY pgs=10926 cs=0 l=1 rev1=1 rx=0x7f21cc041010 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:11.801 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:11.797+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37307 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:11.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:20:11.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc009810 con 0x7f21cc0138f0 2021-03-09T10:20:11.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.801+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:20:11.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.801+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:11.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.801+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:11.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:11.801+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:16.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc027740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:16.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc027740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:16.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc027740 secure :-1 s=READY pgs=10926 cs=0 l=1 rev1=1 rx=0x7f21cc041010 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:16.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc027740 secure :-1 s=READY pgs=10926 cs=0 l=1 rev1=1 rx=0x7f21cc041010 tx=0x7f21dc02dfc0).stop 2021-03-09T10:20:16.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc027740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:16.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc027740 unknown :-1 s=CLOSED pgs=10926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:16.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:16.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:16.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:16.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d0d0 con 0x7f21cc005e30 2021-03-09T10:20:16.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21c4003cb0 2021-03-09T10:20:16.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027c30 con 0x7f21cc042f20 2021-03-09T10:20:16.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:16.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:16.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:16.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:16.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:16.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:16.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:16.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:16.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:16.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc042f20 2021-03-09T10:20:16.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21cc042f20 2021-03-09T10:20:16.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.796+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10936 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e007d630).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:16.808 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:16.800+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37346 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:16.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc042f20 2021-03-09T10:20:16.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc042f20 2021-03-09T10:20:16.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:20:16.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.804+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:16.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.808+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:16.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:16.808+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:21.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:21.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:21.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10936 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e007d630).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:21.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10936 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e007d630).stop 2021-03-09T10:20:21.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:21.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:21.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:21.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:21.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:21.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237f0 con 0x7f21cc005e30 2021-03-09T10:20:21.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21c4003cb0 2021-03-09T10:20:21.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016a00 con 0x7f21cc0138f0 2021-03-09T10:20:21.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:21.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:21.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:21.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:21.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:21.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:21.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:21.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:21.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:21.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463f0 con 0x7f21cc0138f0 2021-03-09T10:20:21.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21cc0138f0 2021-03-09T10:20:21.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084cb0 secure :-1 s=READY pgs=10947 cs=0 l=1 rev1=1 rx=0x7f21e0081aa0 tx=0x7f21e00816c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:21.815 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:21.808+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37382 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:21.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0030 con 0x7f21cc0138f0 2021-03-09T10:20:21.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fc40 con 0x7f21cc0138f0 2021-03-09T10:20:21.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00078d0 con 0x7f21cc0138f0 2021-03-09T10:20:21.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.812+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:21.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.812+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:21.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:21.812+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:25.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:25.797 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:26.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:20:26.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:26.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084cb0 secure :-1 s=READY pgs=10947 cs=0 l=1 rev1=1 rx=0x7f21e0081aa0 tx=0x7f21e00816c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:26.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084cb0 secure :-1 s=READY pgs=10947 cs=0 l=1 rev1=1 rx=0x7f21e0081aa0 tx=0x7f21e00816c0).stop 2021-03-09T10:20:26.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:26.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084cb0 unknown :-1 s=CLOSED pgs=10947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:26.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:26.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:26.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:26.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237f0 con 0x7f21cc005e30 2021-03-09T10:20:26.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013470 con 0x7f21c4003cb0 2021-03-09T10:20:26.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d070 con 0x7f21cc042f20 2021-03-09T10:20:26.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:26.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:26.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:26.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:26.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:26.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:26.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:26.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:26.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:26.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016a00 con 0x7f21cc042f20 2021-03-09T10:20:26.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0132e0 con 0x7f21cc042f20 2021-03-09T10:20:26.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10958 cs=0 l=1 rev1=1 rx=0x7f21cc017740 tx=0x7f21e4024000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:26.822 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:26.816+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37421 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:26.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T10:20:26.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T10:20:26.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T10:20:26.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.824+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:26.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:26.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:26.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:31.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:31.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:31.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10958 cs=0 l=1 rev1=1 rx=0x7f21cc017740 tx=0x7f21e4024000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:31.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10958 cs=0 l=1 rev1=1 rx=0x7f21cc017740 tx=0x7f21e4024000).stop 2021-03-09T10:20:31.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:31.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:31.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:31.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:31.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:31.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096e50 con 0x7f21cc005e30 2021-03-09T10:20:31.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d070 con 0x7f21c4003cb0 2021-03-09T10:20:31.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d180 con 0x7f21cc0138f0 2021-03-09T10:20:31.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:31.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:31.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:31.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:31.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:31.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:31.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:31.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:31.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:31.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025f50 con 0x7f21cc0138f0 2021-03-09T10:20:31.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016a00 con 0x7f21cc0138f0 2021-03-09T10:20:31.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=10968 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4003bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:31.829 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:31.824+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37457 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:31.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f210 con 0x7f21cc0138f0 2021-03-09T10:20:31.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401a420 con 0x7f21cc0138f0 2021-03-09T10:20:31.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:20:31.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.828+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:31.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.828+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:31.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:31.828+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:36.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:36.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:36.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=10968 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4003bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:36.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=10968 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e4003bf0).stop 2021-03-09T10:20:36.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:36.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=10968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:36.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:36.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:36.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:36.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096e50 con 0x7f21cc005e30 2021-03-09T10:20:36.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f50 con 0x7f21c4003cb0 2021-03-09T10:20:36.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ee20 con 0x7f21cc042f20 2021-03-09T10:20:36.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:36.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:36.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:36.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:36.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:36.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:36.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:36.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:36.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:36.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ef60 con 0x7f21cc042f20 2021-03-09T10:20:36.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084b20 con 0x7f21cc042f20 2021-03-09T10:20:36.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10979 cs=0 l=1 rev1=1 rx=0x7f21dc005a60 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:36.837 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:36.832+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37496 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:36.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:20:36.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.832+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0535a0 con 0x7f21cc042f20 2021-03-09T10:20:36.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:20:36.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.836+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:36.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.836+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:36.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:36.836+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:41.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:41.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:41.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10979 cs=0 l=1 rev1=1 rx=0x7f21dc005a60 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:41.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=10979 cs=0 l=1 rev1=1 rx=0x7f21dc005a60 tx=0x7f21dc010d90).stop 2021-03-09T10:20:41.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:41.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=10979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:41.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:41.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:41.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:41.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ee20 con 0x7f21cc005e30 2021-03-09T10:20:41.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:20:41.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc0138f0 2021-03-09T10:20:41.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:41.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:41.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:41.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:41.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:41.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:41.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:41.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:41.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:41.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096c60 con 0x7f21cc0138f0 2021-03-09T10:20:41.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ef60 con 0x7f21cc0138f0 2021-03-09T10:20:41.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.836+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=10990 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0abef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:41.845 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:41.840+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37532 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:41.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc0138f0 2021-03-09T10:20:41.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc0138f0 2021-03-09T10:20:41.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0138f0 2021-03-09T10:20:41.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.844+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:41.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:41.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:41.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:46.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:46.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:46.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=10990 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0abef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:46.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=10990 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0abef0).stop 2021-03-09T10:20:46.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:46.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=10990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:46.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:46.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:46.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:46.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f550 con 0x7f21cc005e30 2021-03-09T10:20:46.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:20:46.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc042f20 2021-03-09T10:20:46.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:46.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:46.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:46.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:46.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:46.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:46.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:46.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:46.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:46.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02ee20 con 0x7f21cc042f20 2021-03-09T10:20:46.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016a00 con 0x7f21cc042f20 2021-03-09T10:20:46.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.844+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11002 cs=0 l=1 rev1=1 rx=0x7f21e4036330 tx=0x7f21e400f3d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:46.852 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:46.848+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37571 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:46.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401a350 con 0x7f21cc042f20 2021-03-09T10:20:46.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401a350 con 0x7f21cc042f20 2021-03-09T10:20:46.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:20:46.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.860+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:46.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.860+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:46.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:46.860+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:51.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:51.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:51.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11002 cs=0 l=1 rev1=1 rx=0x7f21e4036330 tx=0x7f21e400f3d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:51.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11002 cs=0 l=1 rev1=1 rx=0x7f21e4036330 tx=0x7f21e400f3d0).stop 2021-03-09T10:20:51.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:51.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=11002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:51.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:51.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:51.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:51.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc005e30 2021-03-09T10:20:51.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f6f0 con 0x7f21c4003cb0 2021-03-09T10:20:51.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f150 con 0x7f21cc0138f0 2021-03-09T10:20:51.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:51.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:51.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:51.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:51.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:51.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:51.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:51.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:51.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:51.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084b20 con 0x7f21cc0138f0 2021-03-09T10:20:51.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096c60 con 0x7f21cc0138f0 2021-03-09T10:20:51.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.852+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11011 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc080040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:51.861 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:51.856+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37607 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:51.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0330c0 con 0x7f21cc0138f0 2021-03-09T10:20:51.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21cc0138f0 2021-03-09T10:20:51.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:20:51.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.860+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:51.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.864+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:51.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:51.864+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:20:55.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:55.845 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:20:56.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:56.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:56.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11011 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc080040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:56.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11011 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc080040).stop 2021-03-09T10:20:56.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:56.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=11011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:56.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:56.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:56.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:56.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0409f0 con 0x7f21cc005e30 2021-03-09T10:20:56.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4003cb0 2021-03-09T10:20:56.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc042f20 2021-03-09T10:20:56.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:20:56.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:56.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:20:56.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:56.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.856+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:56.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:56.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:56.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:56.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:56.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096c60 con 0x7f21cc042f20 2021-03-09T10:20:56.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21cc042f20 2021-03-09T10:20:56.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.860+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11023 cs=0 l=1 rev1=1 rx=0x7f21e0003550 tx=0x7f21e003d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:56.868 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:20:56.864+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37646 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:20:56.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.864+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc042f20 2021-03-09T10:20:56.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.872+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc042f20 2021-03-09T10:20:56.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.872+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:20:56.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.872+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:20:56.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.872+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:20:56.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:20:56.872+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:01.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:01.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:01.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11023 cs=0 l=1 rev1=1 rx=0x7f21e0003550 tx=0x7f21e003d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:01.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11023 cs=0 l=1 rev1=1 rx=0x7f21e0003550 tx=0x7f21e003d2e0).stop 2021-03-09T10:21:01.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:01.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=11023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:01.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:01.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:01.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:01.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc005e30 2021-03-09T10:21:01.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040800 con 0x7f21c4003cb0 2021-03-09T10:21:01.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040940 con 0x7f21cc0138f0 2021-03-09T10:21:01.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:01.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:01.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:01.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:01.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:01.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:01.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:01.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:01.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:01.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084cb0 con 0x7f21cc0138f0 2021-03-09T10:21:01.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02ee20 con 0x7f21cc0138f0 2021-03-09T10:21:01.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.867+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11038 cs=0 l=1 rev1=1 rx=0x7f21cc01b0e0 tx=0x7f21e4024480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:01.876 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:01.871+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37682 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:01.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f3a0 con 0x7f21cc0138f0 2021-03-09T10:21:01.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f3a0 con 0x7f21cc0138f0 2021-03-09T10:21:01.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:21:01.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.875+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:01.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.879+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:01.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:01.879+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:06.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:06.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:06.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11038 cs=0 l=1 rev1=1 rx=0x7f21cc01b0e0 tx=0x7f21e4024480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:06.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11038 cs=0 l=1 rev1=1 rx=0x7f21cc01b0e0 tx=0x7f21e4024480).stop 2021-03-09T10:21:06.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:06.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=11038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:06.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:06.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:06.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:06.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc005e30 2021-03-09T10:21:06.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ec30 con 0x7f21c4003cb0 2021-03-09T10:21:06.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ed70 con 0x7f21cc042f20 2021-03-09T10:21:06.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:06.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:06.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:06.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:06.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:06.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:06.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:06.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:06.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:06.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02eeb0 con 0x7f21cc042f20 2021-03-09T10:21:06.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084b20 con 0x7f21cc042f20 2021-03-09T10:21:06.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11043 cs=0 l=1 rev1=1 rx=0x7f21cc045270 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:06.884 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:06.875+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37721 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:06.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc020480 con 0x7f21cc042f20 2021-03-09T10:21:06.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.879+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:21:06.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:21:06.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.883+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:06.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.883+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:06.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:06.883+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:11.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:11.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:11.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11043 cs=0 l=1 rev1=1 rx=0x7f21cc045270 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:11.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11043 cs=0 l=1 rev1=1 rx=0x7f21cc045270 tx=0x7f21dc010d20).stop 2021-03-09T10:21:11.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:11.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=11043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:11.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:11.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:11.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:11.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc005e30 2021-03-09T10:21:11.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084b20 con 0x7f21c4003cb0 2021-03-09T10:21:11.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:21:11.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:11.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:11.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:11.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:11.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:11.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:11.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:11.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:11.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:11.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084e40 con 0x7f21cc0138f0 2021-03-09T10:21:11.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.883+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084cb0 con 0x7f21cc0138f0 2021-03-09T10:21:11.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.887+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11054 cs=0 l=1 rev1=1 rx=0x7f21cc012500 tx=0x7f21e4036250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:11.891 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:11.887+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37757 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:11.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017350 con 0x7f21cc0138f0 2021-03-09T10:21:11.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc0138f0 2021-03-09T10:21:11.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:21:11.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.891+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:11.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.895+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:11.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:11.895+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:16.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.887+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:16.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.887+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:16.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.887+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11054 cs=0 l=1 rev1=1 rx=0x7f21cc012500 tx=0x7f21e4036250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:16.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.887+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11054 cs=0 l=1 rev1=1 rx=0x7f21cc012500 tx=0x7f21e4036250).stop 2021-03-09T10:21:16.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.887+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:16.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.887+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=11054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:16.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:16.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:16.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:16.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0947a0 con 0x7f21cc005e30 2021-03-09T10:21:16.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21c4003cb0 2021-03-09T10:21:16.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084cc0 con 0x7f21cc042f20 2021-03-09T10:21:16.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:16.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:16.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:16.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:16.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:16.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:16.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:16.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:16.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:16.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084fd0 con 0x7f21cc042f20 2021-03-09T10:21:16.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02eeb0 con 0x7f21cc042f20 2021-03-09T10:21:16.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11064 cs=0 l=1 rev1=1 rx=0x7f21cc013010 tx=0x7f21dc0529d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:16.899 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:16.895+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37796 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:16.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:21:16.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc052e30 con 0x7f21cc042f20 2021-03-09T10:21:16.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc042f20 2021-03-09T10:21:16.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.903+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:16.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:16.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:16.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:21.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:21.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.895+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:21.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11064 cs=0 l=1 rev1=1 rx=0x7f21cc013010 tx=0x7f21dc0529d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:21.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11064 cs=0 l=1 rev1=1 rx=0x7f21cc013010 tx=0x7f21dc0529d0).stop 2021-03-09T10:21:21.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.895+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:21.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=11064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:21.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:21.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:21.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:21.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0947a0 con 0x7f21cc005e30 2021-03-09T10:21:21.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac50 con 0x7f21c4003cb0 2021-03-09T10:21:21.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc0138f0 2021-03-09T10:21:21.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:21.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:21.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:21.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:21.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:21.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:21.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:21.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:21.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:21.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cdf0 con 0x7f21cc0138f0 2021-03-09T10:21:21.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084fd0 con 0x7f21cc0138f0 2021-03-09T10:21:21.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ec30 secure :-1 s=READY pgs=11075 cs=0 l=1 rev1=1 rx=0x7f21cc02d520 tx=0x7f21dc020550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:21.912 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:21.903+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37832 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:21.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b61d0 con 0x7f21cc0138f0 2021-03-09T10:21:21.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b61d0 con 0x7f21cc0138f0 2021-03-09T10:21:21.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc0138f0 2021-03-09T10:21:21.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.915+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:21.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.915+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:21.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:21.915+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:25.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:25.892 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:26.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:26.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:26.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ec30 secure :-1 s=READY pgs=11075 cs=0 l=1 rev1=1 rx=0x7f21cc02d520 tx=0x7f21dc020550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:26.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ec30 secure :-1 s=READY pgs=11075 cs=0 l=1 rev1=1 rx=0x7f21cc02d520 tx=0x7f21dc020550).stop 2021-03-09T10:21:26.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc02ec30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:26.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc02ec30 unknown :-1 s=CLOSED pgs=11075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:26.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.903+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:26.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:26.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:26.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21cc005e30 2021-03-09T10:21:26.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21c4003cb0 2021-03-09T10:21:26.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc084cc0 con 0x7f21cc042f20 2021-03-09T10:21:26.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:26.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:26.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:26.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:26.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:26.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:26.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:26.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:26.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:26.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04cf10 con 0x7f21cc042f20 2021-03-09T10:21:26.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084e40 con 0x7f21cc042f20 2021-03-09T10:21:26.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.907+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11086 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4016b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:26.915 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:26.915+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37871 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:26.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40057e0 con 0x7f21cc042f20 2021-03-09T10:21:26.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40057e0 con 0x7f21cc042f20 2021-03-09T10:21:26.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc042f20 2021-03-09T10:21:26.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.915+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:26.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.919+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:26.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:26.919+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:31.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:31.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:31.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11086 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4016b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:31.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 secure :-1 s=READY pgs=11086 cs=0 l=1 rev1=1 rx=0x7f21e40172e0 tx=0x7f21e4016b20).stop 2021-03-09T10:21:31.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc012b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:31.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc012b20 unknown :-1 s=CLOSED pgs=11086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:31.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:31.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:31.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:31.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:21:31.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096b10 con 0x7f21c4003cb0 2021-03-09T10:21:31.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:21:31.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:31.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:31.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:31.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:31.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.915+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:31.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:31.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:31.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:31.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:31.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084b20 con 0x7f21cc0138f0 2021-03-09T10:21:31.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096c60 con 0x7f21cc0138f0 2021-03-09T10:21:31.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.919+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 secure :-1 s=READY pgs=11098 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e0013d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:31.925 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:31.923+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37907 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:31.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.923+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0138f0 2021-03-09T10:21:31.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0138f0 2021-03-09T10:21:31.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003d010 con 0x7f21cc0138f0 2021-03-09T10:21:31.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.927+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:31.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.927+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:31.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:31.927+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:36.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:36.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:36.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 secure :-1 s=READY pgs=11098 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e0013d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:36.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 secure :-1 s=READY pgs=11098 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e0013d60).stop 2021-03-09T10:21:36.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04d0a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:36.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 unknown :-1 s=CLOSED pgs=11098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:36.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:36.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:36.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:36.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:21:36.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0176f0 con 0x7f21c4003cb0 2021-03-09T10:21:36.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:21:36.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:36.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:36.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:36.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:36.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.923+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:36.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:36.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:36.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:36.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:36.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084cb0 con 0x7f21cc042f20 2021-03-09T10:21:36.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cdf0 con 0x7f21cc042f20 2021-03-09T10:21:36.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096940 secure :-1 s=READY pgs=11109 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0871f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:36.937 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:36.927+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37946 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:36.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:21:36.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:21:36.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:21:36.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.935+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:36.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.935+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:36.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:36.935+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:41.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:41.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:41.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096940 secure :-1 s=READY pgs=11109 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0871f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:41.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096940 secure :-1 s=READY pgs=11109 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0871f0).stop 2021-03-09T10:21:41.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:41.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096940 unknown :-1 s=CLOSED pgs=11109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:41.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:41.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:41.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:41.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:21:41.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041740 con 0x7f21c4003cb0 2021-03-09T10:21:41.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21cc0138f0 2021-03-09T10:21:41.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:41.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:41.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:41.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:41.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:41.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:41.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:41.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:41.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:41.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cdf0 con 0x7f21cc0138f0 2021-03-09T10:21:41.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04cf10 con 0x7f21cc0138f0 2021-03-09T10:21:41.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.935+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 secure :-1 s=READY pgs=11119 cs=0 l=1 rev1=1 rx=0x7f21e4028f50 tx=0x7f21e4007160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:41.939 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:41.939+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37982 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:41.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.939+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407fd30 con 0x7f21cc0138f0 2021-03-09T10:21:41.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.943+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024410 con 0x7f21cc0138f0 2021-03-09T10:21:41.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.947+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:21:41.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.947+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:41.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.947+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:41.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:41.947+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:46.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:46.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:46.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 secure :-1 s=READY pgs=11119 cs=0 l=1 rev1=1 rx=0x7f21e4028f50 tx=0x7f21e4007160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:46.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 secure :-1 s=READY pgs=11119 cs=0 l=1 rev1=1 rx=0x7f21e4028f50 tx=0x7f21e4007160).stop 2021-03-09T10:21:46.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04d0a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:46.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04d0a0 unknown :-1 s=CLOSED pgs=11119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:46.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:46.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:46.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:46.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:21:46.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21c4003cb0 2021-03-09T10:21:46.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04cf10 con 0x7f21cc042f20 2021-03-09T10:21:46.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:46.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:46.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:46.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:46.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.939+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:46.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:46.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:46.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:46.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:46.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:21:46.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084cb0 con 0x7f21cc042f20 2021-03-09T10:21:46.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.943+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096920 secure :-1 s=READY pgs=11129 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc081090).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:46.948 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:46.947+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38021 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:46.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.947+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00aee0 con 0x7f21cc042f20 2021-03-09T10:21:46.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.951+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00aee0 con 0x7f21cc042f20 2021-03-09T10:21:46.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.951+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc042f20 2021-03-09T10:21:46.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.951+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:46.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.951+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:46.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:46.951+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:51.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:51.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:51.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096920 secure :-1 s=READY pgs=11129 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc081090).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:51.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096920 secure :-1 s=READY pgs=11129 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc081090).stop 2021-03-09T10:21:51.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc096920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:51.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc096920 unknown :-1 s=CLOSED pgs=11129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:51.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:51.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:51.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:51.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:21:51.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21c4003cb0 2021-03-09T10:21:51.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:21:51.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:51.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:51.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:51.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:51.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:51.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:51.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:51.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:51.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:51.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084cb0 con 0x7f21cc0138f0 2021-03-09T10:21:51.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:21:51.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.950+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11140 cs=0 l=1 rev1=1 rx=0x7f21dc0629c0 tx=0x7f21dc009890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:51.957 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:51.954+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38057 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:51.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0138f0 2021-03-09T10:21:51.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21cc0138f0 2021-03-09T10:21:51.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0138f0 2021-03-09T10:21:51.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.962+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:51.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.962+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:51.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:51.962+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:21:55.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:55.941 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:21:56.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:56.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:56.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11140 cs=0 l=1 rev1=1 rx=0x7f21dc0629c0 tx=0x7f21dc009890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:56.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11140 cs=0 l=1 rev1=1 rx=0x7f21dc0629c0 tx=0x7f21dc009890).stop 2021-03-09T10:21:56.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:56.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=11140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:56.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:56.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:56.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:56.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:21:56.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21c4003cb0 2021-03-09T10:21:56.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21cc042f20 2021-03-09T10:21:56.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:21:56.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:56.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:21:56.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:56.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:56.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:56.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:56.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:56.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:56.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d400 con 0x7f21cc042f20 2021-03-09T10:21:56.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cdf0 con 0x7f21cc042f20 2021-03-09T10:21:56.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084e40 secure :-1 s=READY pgs=11150 cs=0 l=1 rev1=1 rx=0x7f21cc02f290 tx=0x7f21e40a5250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:56.969 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:21:56.962+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38096 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:21:56.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:21:56.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:21:56.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:21:56.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.970+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:21:56.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.970+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:21:56.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:21:56.970+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:01.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:01.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:01.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084e40 secure :-1 s=READY pgs=11150 cs=0 l=1 rev1=1 rx=0x7f21cc02f290 tx=0x7f21e40a5250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:01.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084e40 secure :-1 s=READY pgs=11150 cs=0 l=1 rev1=1 rx=0x7f21cc02f290 tx=0x7f21e40a5250).stop 2021-03-09T10:22:01.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:01.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084e40 unknown :-1 s=CLOSED pgs=11150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:01.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:01.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:01.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:01.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:01.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:01.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21c4003cb0 2021-03-09T10:22:01.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028030 con 0x7f21cc0138f0 2021-03-09T10:22:01.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:01.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:01.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:01.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.962+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:01.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:01.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:01.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:01.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:01.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:22:01.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084cb0 con 0x7f21cc0138f0 2021-03-09T10:22:01.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11165 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:01.972 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:01.970+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38132 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:01.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bbf00 con 0x7f21cc0138f0 2021-03-09T10:22:01.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.974+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0bbf00 con 0x7f21cc0138f0 2021-03-09T10:22:01.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:22:01.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.978+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:01.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.978+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:01.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:01.978+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:06.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:06.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:06.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11165 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:06.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11165 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc02dfc0).stop 2021-03-09T10:22:06.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:06.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=11165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:06.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:06.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:06.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:06.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:06.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21c4003cb0 2021-03-09T10:22:06.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046160 con 0x7f21cc042f20 2021-03-09T10:22:06.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:06.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:06.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:06.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:06.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:06.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:06.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:06.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:06.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:06.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cdf0 con 0x7f21cc042f20 2021-03-09T10:22:06.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:22:06.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.974+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cb0 secure :-1 s=READY pgs=11171 cs=0 l=1 rev1=1 rx=0x7f21dc057a50 tx=0x7f21dc052070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:06.980 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:06.978+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38171 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:06.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc021e80 con 0x7f21cc042f20 2021-03-09T10:22:06.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc021e80 con 0x7f21cc042f20 2021-03-09T10:22:06.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.982+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc057c50 con 0x7f21cc042f20 2021-03-09T10:22:06.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.982+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:06.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.982+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:06.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:06.982+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:11.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:22:11.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:11.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cb0 secure :-1 s=READY pgs=11171 cs=0 l=1 rev1=1 rx=0x7f21dc057a50 tx=0x7f21dc052070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:11.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cb0 secure :-1 s=READY pgs=11171 cs=0 l=1 rev1=1 rx=0x7f21dc057a50 tx=0x7f21dc052070).stop 2021-03-09T10:22:11.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:11.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084cb0 unknown :-1 s=CLOSED pgs=11171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:11.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:11.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:11.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:11.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028030 con 0x7f21cc005e30 2021-03-09T10:22:11.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21c4003cb0 2021-03-09T10:22:11.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21cc0138f0 2021-03-09T10:22:11.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:11.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:11.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:11.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:11.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:11.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:11.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:11.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:11.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:11.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:22:11.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084b20 con 0x7f21cc0138f0 2021-03-09T10:22:11.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.982+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11183 cs=0 l=1 rev1=1 rx=0x7f21e00a0810 tx=0x7f21e00a0730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:11.987 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:11.986+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38207 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:11.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:22:11.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:22:11.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:22:11.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.994+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:11.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.994+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:11.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:11.994+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:16.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:16.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:16.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11183 cs=0 l=1 rev1=1 rx=0x7f21e00a0810 tx=0x7f21e00a0730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:16.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 secure :-1 s=READY pgs=11183 cs=0 l=1 rev1=1 rx=0x7f21e00a0810 tx=0x7f21e00a0730).stop 2021-03-09T10:22:16.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04cf10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:16.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04cf10 unknown :-1 s=CLOSED pgs=11183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:16.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:16.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:16.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:16.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046160 con 0x7f21cc005e30 2021-03-09T10:22:16.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21c4003cb0 2021-03-09T10:22:16.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21cc042f20 2021-03-09T10:22:16.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:16.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:16.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:16.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:16.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:16.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:16.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:16.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:16.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:16.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058920 con 0x7f21cc042f20 2021-03-09T10:22:16.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cdf0 con 0x7f21cc042f20 2021-03-09T10:22:16.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11192 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc015a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:16.997 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:16.994+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38246 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:16.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:22:16.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc085bb0 con 0x7f21cc042f20 2021-03-09T10:22:16.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:22:16.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.998+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:17.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:17.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:16.998+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:21.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:22:21.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:21.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11192 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc015a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:21.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11192 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc015a90).stop 2021-03-09T10:22:21.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:21.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=CLOSED pgs=11192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:21.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:21.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:21.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:21.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028030 con 0x7f21cc005e30 2021-03-09T10:22:22.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21c4003cb0 2021-03-09T10:22:22.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21cc0138f0 2021-03-09T10:22:22.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:22.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:22.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:22.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:22.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.994+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:22.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:22.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:22.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:22.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:22.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21cc0138f0 2021-03-09T10:22:22.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:22:22.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:21.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cab0 secure :-1 s=READY pgs=11204 cs=0 l=1 rev1=1 rx=0x7f21e0098770 tx=0x7f21e0075c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:22.004 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:22.002+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38282 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:22.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:22.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0013dd0 con 0x7f21cc0138f0 2021-03-09T10:22:22.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:22.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21cc0138f0 2021-03-09T10:22:22.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:22.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:22:22.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:22.006+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:22.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:22.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:22.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:22.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:25.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:25.991 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:27.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:27.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:27.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cab0 secure :-1 s=READY pgs=11204 cs=0 l=1 rev1=1 rx=0x7f21e0098770 tx=0x7f21e0075c90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:27.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cab0 secure :-1 s=READY pgs=11204 cs=0 l=1 rev1=1 rx=0x7f21e0098770 tx=0x7f21e0075c90).stop 2021-03-09T10:22:27.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01cab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:27.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01cab0 unknown :-1 s=CLOSED pgs=11204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:27.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:27.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:27.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:27.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046160 con 0x7f21cc005e30 2021-03-09T10:22:27.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:27.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21c4003cb0 2021-03-09T10:22:27.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21cc042f20 2021-03-09T10:22:27.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:27.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:27.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:27.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:27.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:27.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:27.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:27.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:27.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:22:27.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.002+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d400 con 0x7f21cc042f20 2021-03-09T10:22:27.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.006+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11214 cs=0 l=1 rev1=1 rx=0x7f21e400f310 tx=0x7f21e4036f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:27.012 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:27.010+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38321 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:27.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005110 con 0x7f21cc042f20 2021-03-09T10:22:27.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4059000 con 0x7f21cc042f20 2021-03-09T10:22:27.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:22:27.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.014+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:27.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.014+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:27.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:27.014+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:32.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:32.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:32.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11214 cs=0 l=1 rev1=1 rx=0x7f21e400f310 tx=0x7f21e4036f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:32.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11214 cs=0 l=1 rev1=1 rx=0x7f21e400f310 tx=0x7f21e4036f10).stop 2021-03-09T10:22:32.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:32.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=CLOSED pgs=11214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:32.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:32.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:32.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:32.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028030 con 0x7f21cc005e30 2021-03-09T10:22:32.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21c4003cb0 2021-03-09T10:22:32.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21cc0138f0 2021-03-09T10:22:32.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:32.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:32.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:32.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:32.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.010+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:32.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:32.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:32.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:32.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:32.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:22:32.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058920 con 0x7f21cc0138f0 2021-03-09T10:22:32.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11225 cs=0 l=1 rev1=1 rx=0x7f21dc0ba510 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:32.021 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:32.018+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38357 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:32.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062300 con 0x7f21cc0138f0 2021-03-09T10:22:32.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21cc0138f0 2021-03-09T10:22:32.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc0138f0 2021-03-09T10:22:32.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.026+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:32.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:32.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:32.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:37.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:22:37.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:37.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11225 cs=0 l=1 rev1=1 rx=0x7f21dc0ba510 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:37.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11225 cs=0 l=1 rev1=1 rx=0x7f21dc0ba510 tx=0x7f21dc010d90).stop 2021-03-09T10:22:37.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:37.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=CLOSED pgs=11225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:37.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:37.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:37.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:37.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:37.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21c4003cb0 2021-03-09T10:22:37.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a460 con 0x7f21cc042f20 2021-03-09T10:22:37.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:37.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:37.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:37.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:37.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:37.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:37.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:37.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:37.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:37.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058920 con 0x7f21cc042f20 2021-03-09T10:22:37.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21cc042f20 2021-03-09T10:22:37.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.018+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11235 cs=0 l=1 rev1=1 rx=0x7f21e00a6340 tx=0x7f21e00a3650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:37.029 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:37.026+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38396 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:37.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a3880 con 0x7f21cc042f20 2021-03-09T10:22:37.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21cc042f20 2021-03-09T10:22:37.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a3d80 con 0x7f21cc042f20 2021-03-09T10:22:37.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.030+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:37.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.030+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:37.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:37.030+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:42.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:42.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:42.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11235 cs=0 l=1 rev1=1 rx=0x7f21e00a6340 tx=0x7f21e00a3650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:42.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 secure :-1 s=READY pgs=11235 cs=0 l=1 rev1=1 rx=0x7f21e00a6340 tx=0x7f21e00a3650).stop 2021-03-09T10:22:42.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:42.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084b20 unknown :-1 s=CLOSED pgs=11235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:42.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:42.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:42.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:42.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:42.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a690 con 0x7f21c4003cb0 2021-03-09T10:22:42.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036bc0 con 0x7f21cc0138f0 2021-03-09T10:22:42.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:42.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:42.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:42.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:42.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:42.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:42.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:42.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:42.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:42.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ced0 con 0x7f21cc0138f0 2021-03-09T10:22:42.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:22:42.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.026+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11246 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e408fa00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:42.035 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:42.034+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:42.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:22:42.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc0138f0 2021-03-09T10:22:42.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:22:42.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.034+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:42.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.038+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:42.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:42.038+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:47.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:47.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:47.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11246 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e408fa00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:47.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11246 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e408fa00).stop 2021-03-09T10:22:47.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:47.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=CLOSED pgs=11246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:47.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:47.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:47.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:47.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:47.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f920 con 0x7f21c4003cb0 2021-03-09T10:22:47.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fd0 con 0x7f21cc042f20 2021-03-09T10:22:47.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:47.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:47.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:47.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:47.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:47.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:47.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:47.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:47.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:47.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc028030 con 0x7f21cc042f20 2021-03-09T10:22:47.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ced0 con 0x7f21cc042f20 2021-03-09T10:22:47.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.037+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a1b0 secure :-1 s=READY pgs=11256 cs=0 l=1 rev1=1 rx=0x7f21cc030de0 tx=0x7f21e409c0e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:47.044 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:47.037+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38471 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:47.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.041+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:22:47.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400f310 con 0x7f21cc042f20 2021-03-09T10:22:47.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc042f20 2021-03-09T10:22:47.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.049+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:47.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.049+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:47.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:47.049+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:52.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:52.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:52.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a1b0 secure :-1 s=READY pgs=11256 cs=0 l=1 rev1=1 rx=0x7f21cc030de0 tx=0x7f21e409c0e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:52.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a1b0 secure :-1 s=READY pgs=11256 cs=0 l=1 rev1=1 rx=0x7f21cc030de0 tx=0x7f21e409c0e0).stop 2021-03-09T10:22:52.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a1b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:52.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a1b0 unknown :-1 s=CLOSED pgs=11256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:52.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:52.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:52.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:52.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:52.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fd0 con 0x7f21c4003cb0 2021-03-09T10:22:52.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f920 con 0x7f21cc0138f0 2021-03-09T10:22:52.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:52.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:52.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:52.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:52.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:52.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:52.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:52.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:52.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:52.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21cc0138f0 2021-03-09T10:22:52.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc028030 con 0x7f21cc0138f0 2021-03-09T10:22:52.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11267 cs=0 l=1 rev1=1 rx=0x7f21cc08d7e0 tx=0x7f21e4020f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:52.051 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:52.045+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38507 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:52.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402d410 con 0x7f21cc0138f0 2021-03-09T10:22:52.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402d230 con 0x7f21cc0138f0 2021-03-09T10:22:52.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e401a270 con 0x7f21cc0138f0 2021-03-09T10:22:52.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.053+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:52.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.053+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:52.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:52.053+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:22:56.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:56.039 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:22:57.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:57.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:57.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11267 cs=0 l=1 rev1=1 rx=0x7f21cc08d7e0 tx=0x7f21e4020f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:57.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11267 cs=0 l=1 rev1=1 rx=0x7f21cc08d7e0 tx=0x7f21e4020f60).stop 2021-03-09T10:22:57.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:57.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=CLOSED pgs=11267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:57.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:57.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc029e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:57.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:57.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:22:57.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fd0 con 0x7f21c4003cb0 2021-03-09T10:22:57.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:22:57.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bc0 con 0x7f21cc042f20 2021-03-09T10:22:57.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:57.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:22:57.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:57.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc029e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:57.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:57.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:57.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:57.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:57.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:22:57.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21cc042f20 2021-03-09T10:22:57.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc029e70 secure :-1 s=READY pgs=11278 cs=0 l=1 rev1=1 rx=0x7f21e404f640 tx=0x7f21e402b200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:57.059 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:22:57.053+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38546 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:22:57.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402d410 con 0x7f21cc042f20 2021-03-09T10:22:57.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402d410 con 0x7f21cc042f20 2021-03-09T10:22:57.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:22:57.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.061+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:22:57.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.061+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:22:57.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:22:57.061+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:02.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc029e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:02.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc029e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:02.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc029e70 secure :-1 s=READY pgs=11278 cs=0 l=1 rev1=1 rx=0x7f21e404f640 tx=0x7f21e402b200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:02.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc029e70 secure :-1 s=READY pgs=11278 cs=0 l=1 rev1=1 rx=0x7f21e404f640 tx=0x7f21e402b200).stop 2021-03-09T10:23:02.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc029e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:02.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc029e70 unknown :-1 s=CLOSED pgs=11278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:02.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:02.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:02.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:02.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:23:02.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fd0 con 0x7f21c4003cb0 2021-03-09T10:23:02.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049bc0 con 0x7f21cc0138f0 2021-03-09T10:23:02.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:02.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:02.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:02.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:02.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:02.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:02.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:02.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:02.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:02.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a410 con 0x7f21cc0138f0 2021-03-09T10:23:02.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:23:02.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11289 cs=0 l=1 rev1=1 rx=0x7f21dc0204c0 tx=0x7f21dc0204f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:02.066 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:02.061+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38582 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:02.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0701f0 con 0x7f21cc0138f0 2021-03-09T10:23:02.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21cc0138f0 2021-03-09T10:23:02.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc0138f0 2021-03-09T10:23:02.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.069+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:02.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.069+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:02.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:02.069+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:07.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:07.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:07.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11289 cs=0 l=1 rev1=1 rx=0x7f21dc0204c0 tx=0x7f21dc0204f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:07.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11289 cs=0 l=1 rev1=1 rx=0x7f21dc0204c0 tx=0x7f21dc0204f0).stop 2021-03-09T10:23:07.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:07.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=CLOSED pgs=11289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:07.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:07.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:07.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:07.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:23:07.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fd0 con 0x7f21c4003cb0 2021-03-09T10:23:07.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057630 con 0x7f21cc042f20 2021-03-09T10:23:07.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:07.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:07.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:07.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:07.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:07.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:07.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:07.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:07.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:07.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058ab0 con 0x7f21cc042f20 2021-03-09T10:23:07.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a410 con 0x7f21cc042f20 2021-03-09T10:23:07.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085500 secure :-1 s=READY pgs=11299 cs=0 l=1 rev1=1 rx=0x7f21cc04c960 tx=0x7f21dc070000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:07.075 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:07.069+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38621 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:07.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01c300 con 0x7f21cc042f20 2021-03-09T10:23:07.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:23:07.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc01c300 con 0x7f21cc042f20 2021-03-09T10:23:07.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.077+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:07.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.077+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:07.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:07.077+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:12.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:12.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:12.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085500 secure :-1 s=READY pgs=11299 cs=0 l=1 rev1=1 rx=0x7f21cc04c960 tx=0x7f21dc070000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:12.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085500 secure :-1 s=READY pgs=11299 cs=0 l=1 rev1=1 rx=0x7f21cc04c960 tx=0x7f21dc070000).stop 2021-03-09T10:23:12.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc085500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:12.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc085500 unknown :-1 s=CLOSED pgs=11299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:12.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:12.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:12.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:12.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:23:12.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027a90 con 0x7f21c4003cb0 2021-03-09T10:23:12.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011fd0 con 0x7f21cc0138f0 2021-03-09T10:23:12.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:12.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:12.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:12.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:12.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.069+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:12.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:12.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:12.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:12.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:12.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:23:12.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058920 con 0x7f21cc0138f0 2021-03-09T10:23:12.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11310 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:12.083 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:12.077+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38657 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:12.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d2a0 con 0x7f21cc0138f0 2021-03-09T10:23:12.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:23:12.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:23:12.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.081+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:12.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:12.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:12.081+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:17.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:17.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:17.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11310 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:17.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 secure :-1 s=READY pgs=11310 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c620).stop 2021-03-09T10:23:17.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc085010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:17.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc085010 unknown :-1 s=CLOSED pgs=11310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:17.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:17.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:17.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:17.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc027ed0 con 0x7f21cc005e30 2021-03-09T10:23:17.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b520 con 0x7f21c4003cb0 2021-03-09T10:23:17.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e10 con 0x7f21cc042f20 2021-03-09T10:23:17.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:17.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:17.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:17.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:17.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:17.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:17.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:17.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:17.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:17.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc058920 con 0x7f21cc042f20 2021-03-09T10:23:17.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:23:17.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11322 cs=0 l=1 rev1=1 rx=0x7f21e40a6560 tx=0x7f21e40b57d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:17.090 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:17.085+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38696 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:17.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:23:17.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400f310 con 0x7f21cc042f20 2021-03-09T10:23:17.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:23:17.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.093+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:17.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:17.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:17.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:22.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:22.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:22.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11322 cs=0 l=1 rev1=1 rx=0x7f21e40a6560 tx=0x7f21e40b57d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:22.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11322 cs=0 l=1 rev1=1 rx=0x7f21e40a6560 tx=0x7f21e40b57d0).stop 2021-03-09T10:23:22.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:22.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=CLOSED pgs=11322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:22.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:22.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:22.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:22.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21cc005e30 2021-03-09T10:23:22.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a410 con 0x7f21c4003cb0 2021-03-09T10:23:22.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017a00 con 0x7f21cc0138f0 2021-03-09T10:23:22.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:22.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:22.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:22.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:22.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:22.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:22.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:22.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:22.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:22.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0408d0 con 0x7f21cc0138f0 2021-03-09T10:23:22.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:23:22.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 secure :-1 s=READY pgs=11332 cs=0 l=1 rev1=1 rx=0x7f21e00591c0 tx=0x7f21e00590b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:22.098 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:22.093+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38732 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:22.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0059460 con 0x7f21cc0138f0 2021-03-09T10:23:22.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:23:22.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:23:22.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.101+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:22.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:22.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:22.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:26.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:26.089 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:27.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc016d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:27.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc016d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:27.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 secure :-1 s=READY pgs=11332 cs=0 l=1 rev1=1 rx=0x7f21e00591c0 tx=0x7f21e00590b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:27.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 secure :-1 s=READY pgs=11332 cs=0 l=1 rev1=1 rx=0x7f21e00591c0 tx=0x7f21e00590b0).stop 2021-03-09T10:23:27.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc016d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:27.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 unknown :-1 s=CLOSED pgs=11332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:27.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:27.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:27.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:27.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a6b0 con 0x7f21cc005e30 2021-03-09T10:23:27.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21c4003cb0 2021-03-09T10:23:27.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017a00 con 0x7f21cc042f20 2021-03-09T10:23:27.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:27.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:27.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:27.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:27.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:27.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:27.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:27.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:27.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:27.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:23:27.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.097+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058ab0 con 0x7f21cc042f20 2021-03-09T10:23:27.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.101+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11345 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0800c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:27.105 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:27.101+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38771 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:27.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:23:27.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:23:27.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc042f20 2021-03-09T10:23:27.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.109+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:27.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.109+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:27.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:27.109+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:32.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:32.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:32.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11345 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0800c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:32.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11345 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0800c0).stop 2021-03-09T10:23:32.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:32.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=CLOSED pgs=11345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:32.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:32.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:32.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:32.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011f80 con 0x7f21cc005e30 2021-03-09T10:23:32.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21c4003cb0 2021-03-09T10:23:32.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ab0 con 0x7f21cc0138f0 2021-03-09T10:23:32.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:32.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:32.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:32.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:32.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.101+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:32.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:32.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:32.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:32.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:32.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a410 con 0x7f21cc0138f0 2021-03-09T10:23:32.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc058920 con 0x7f21cc0138f0 2021-03-09T10:23:32.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.105+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 secure :-1 s=READY pgs=11356 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e404f450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:32.113 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:32.109+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:32.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:23:32.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400f310 con 0x7f21cc0138f0 2021-03-09T10:23:32.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.113+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:23:32.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.113+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:32.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.113+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:32.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:32.113+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:37.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc016d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:37.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc016d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:37.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 secure :-1 s=READY pgs=11356 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e404f450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:37.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 secure :-1 s=READY pgs=11356 cs=0 l=1 rev1=1 rx=0x7f21e40243a0 tx=0x7f21e404f450).stop 2021-03-09T10:23:37.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc016d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:37.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc016d90 unknown :-1 s=CLOSED pgs=11356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:37.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:37.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:37.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:37.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011f80 con 0x7f21cc005e30 2021-03-09T10:23:37.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21c4003cb0 2021-03-09T10:23:37.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21cc042f20 2021-03-09T10:23:37.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:37.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:37.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:37.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:37.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.108+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:37.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:37.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:37.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:37.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:37.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017a00 con 0x7f21cc042f20 2021-03-09T10:23:37.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc042f20 2021-03-09T10:23:37.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.112+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11363 cs=0 l=1 rev1=1 rx=0x7f21e0033840 tx=0x7f21e0075d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:37.121 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:37.116+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38846 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:37.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21cc042f20 2021-03-09T10:23:37.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0033680 con 0x7f21cc042f20 2021-03-09T10:23:37.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:23:37.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.120+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:37.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.120+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:37.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:37.120+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:42.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:42.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:42.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11363 cs=0 l=1 rev1=1 rx=0x7f21e0033840 tx=0x7f21e0075d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:42.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 secure :-1 s=READY pgs=11363 cs=0 l=1 rev1=1 rx=0x7f21e0033840 tx=0x7f21e0075d10).stop 2021-03-09T10:23:42.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc02a210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:42.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc02a210 unknown :-1 s=CLOSED pgs=11363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:42.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:42.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:42.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:42.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011f80 con 0x7f21cc005e30 2021-03-09T10:23:42.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21c4003cb0 2021-03-09T10:23:42.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:23:42.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:42.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:42.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:42.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:42.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:42.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:42.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:42.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:42.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:42.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016d90 con 0x7f21cc0138f0 2021-03-09T10:23:42.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a410 con 0x7f21cc0138f0 2021-03-09T10:23:42.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.120+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11374 cs=0 l=1 rev1=1 rx=0x7f21e4017220 tx=0x7f21e4055d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:42.129 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:42.124+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38882 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:42.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:23:42.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc0138f0 2021-03-09T10:23:42.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:23:42.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.132+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:42.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.132+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:42.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:42.132+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:47.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:47.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:47.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11374 cs=0 l=1 rev1=1 rx=0x7f21e4017220 tx=0x7f21e4055d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:47.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11374 cs=0 l=1 rev1=1 rx=0x7f21e4017220 tx=0x7f21e4055d40).stop 2021-03-09T10:23:47.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:47.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:47.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:47.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:47.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:47.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011f80 con 0x7f21cc005e30 2021-03-09T10:23:47.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097420 con 0x7f21c4003cb0 2021-03-09T10:23:47.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21cc042f20 2021-03-09T10:23:47.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:47.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:47.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:47.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:47.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:47.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:47.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:47.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:47.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:47.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d400 con 0x7f21cc042f20 2021-03-09T10:23:47.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016d90 con 0x7f21cc042f20 2021-03-09T10:23:47.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016f20 secure :-1 s=READY pgs=11384 cs=0 l=1 rev1=1 rx=0x7f21e40b4340 tx=0x7f21e40361e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:47.137 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:47.132+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38921 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:47.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400f240 con 0x7f21cc042f20 2021-03-09T10:23:47.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:23:47.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:23:47.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.136+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:47.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.136+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:47.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:47.136+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:52.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:52.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:52.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016f20 secure :-1 s=READY pgs=11384 cs=0 l=1 rev1=1 rx=0x7f21e40b4340 tx=0x7f21e40361e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:52.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016f20 secure :-1 s=READY pgs=11384 cs=0 l=1 rev1=1 rx=0x7f21e40b4340 tx=0x7f21e40361e0).stop 2021-03-09T10:23:52.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:52.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016f20 unknown :-1 s=CLOSED pgs=11384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:52.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:52.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:52.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:52.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21cc005e30 2021-03-09T10:23:52.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016d90 con 0x7f21c4003cb0 2021-03-09T10:23:52.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012180 con 0x7f21cc0138f0 2021-03-09T10:23:52.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:52.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:52.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:52.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:52.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:52.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:52.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:52.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:52.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:52.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a1f0 con 0x7f21cc0138f0 2021-03-09T10:23:52.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d400 con 0x7f21cc0138f0 2021-03-09T10:23:52.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11395 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e4005f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:52.144 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:52.140+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38956 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:52.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc0138f0 2021-03-09T10:23:52.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.144+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21cc0138f0 2021-03-09T10:23:52.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:23:52.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.148+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:52.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.148+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:52.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:52.148+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:23:56.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:56.154 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:23:57.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:57.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:57.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11395 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e4005f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:57.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11395 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e4005f20).stop 2021-03-09T10:23:57.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:57.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:57.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:57.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:57.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:57.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21cc005e30 2021-03-09T10:23:57.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017280 con 0x7f21c4003cb0 2021-03-09T10:23:57.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012130 con 0x7f21cc042f20 2021-03-09T10:23:57.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:23:57.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:57.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:23:57.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:57.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:57.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:57.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.140+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:57.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.144+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:57.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.144+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:57.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.144+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a380 con 0x7f21cc042f20 2021-03-09T10:23:57.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.144+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017a00 con 0x7f21cc042f20 2021-03-09T10:23:57.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.144+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d90 secure :-1 s=READY pgs=11407 cs=0 l=1 rev1=1 rx=0x7f21e0021000 tx=0x7f21e000b4d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:57.153 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:23:57.148+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38996 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:23:57.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc042f20 2021-03-09T10:23:57.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.152+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc042f20 2021-03-09T10:23:57.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:23:57.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.156+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:23:57.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.156+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:23:57.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:23:57.156+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:02.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:02.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:02.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d90 secure :-1 s=READY pgs=11407 cs=0 l=1 rev1=1 rx=0x7f21e0021000 tx=0x7f21e000b4d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:02.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d90 secure :-1 s=READY pgs=11407 cs=0 l=1 rev1=1 rx=0x7f21e0021000 tx=0x7f21e000b4d0).stop 2021-03-09T10:24:02.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:02.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d90 unknown :-1 s=CLOSED pgs=11407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:02.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:02.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:02.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:02.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0419f0 con 0x7f21cc005e30 2021-03-09T10:24:02.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21c4003cb0 2021-03-09T10:24:02.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0176c0 con 0x7f21cc0138f0 2021-03-09T10:24:02.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:02.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:02.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:02.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:02.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:02.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:02.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:02.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:02.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:02.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017800 con 0x7f21cc0138f0 2021-03-09T10:24:02.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc0138f0 2021-03-09T10:24:02.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.152+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11417 cs=0 l=1 rev1=1 rx=0x7f21cc02ea30 tx=0x7f21dc062190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:02.161 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:02.156+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39031 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:02.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:24:02.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:24:02.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.160+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc0138f0 2021-03-09T10:24:02.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.160+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:02.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.160+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:02.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:02.160+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:07.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:07.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:07.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11417 cs=0 l=1 rev1=1 rx=0x7f21cc02ea30 tx=0x7f21dc062190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:07.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11417 cs=0 l=1 rev1=1 rx=0x7f21cc02ea30 tx=0x7f21dc062190).stop 2021-03-09T10:24:07.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:07.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:07.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:07.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:07.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:07.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0419f0 con 0x7f21cc005e30 2021-03-09T10:24:07.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017690 con 0x7f21c4003cb0 2021-03-09T10:24:07.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21cc042f20 2021-03-09T10:24:07.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:07.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.156+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:07.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:07.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:07.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:07.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:07.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:07.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:07.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.160+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:07.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.164+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:24:07.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.164+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a380 con 0x7f21cc042f20 2021-03-09T10:24:07.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.164+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d40 secure :-1 s=READY pgs=11432 cs=0 l=1 rev1=1 rx=0x7f21e0029ff0 tx=0x7f21e004d6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:07.169 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:07.164+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39071 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:07.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21cc042f20 2021-03-09T10:24:07.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc042f20 2021-03-09T10:24:07.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.172+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:24:07.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.172+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:07.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.172+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:07.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:07.172+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:12.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:12.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:12.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d40 secure :-1 s=READY pgs=11432 cs=0 l=1 rev1=1 rx=0x7f21e0029ff0 tx=0x7f21e004d6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:12.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d40 secure :-1 s=READY pgs=11432 cs=0 l=1 rev1=1 rx=0x7f21e0029ff0 tx=0x7f21e004d6d0).stop 2021-03-09T10:24:12.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:12.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016d40 unknown :-1 s=CLOSED pgs=11432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:12.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:12.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:12.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:12.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:12.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041b90 con 0x7f21cc005e30 2021-03-09T10:24:12.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:12.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d400 con 0x7f21c4003cb0 2021-03-09T10:24:12.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ced0 con 0x7f21cc0138f0 2021-03-09T10:24:12.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:12.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:12.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:12.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:12.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:12.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:12.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:12.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d010 con 0x7f21cc0138f0 2021-03-09T10:24:12.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a1f0 con 0x7f21cc0138f0 2021-03-09T10:24:12.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11438 cs=0 l=1 rev1=1 rx=0x7f21e40a6530 tx=0x7f21e4008e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:12.177 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:12.172+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39106 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:12.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.172+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402bea0 con 0x7f21cc0138f0 2021-03-09T10:24:12.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.172+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024410 con 0x7f21cc0138f0 2021-03-09T10:24:12.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:24:12.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.176+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:12.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.176+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:12.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:12.176+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:17.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:17.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:17.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11438 cs=0 l=1 rev1=1 rx=0x7f21e40a6530 tx=0x7f21e4008e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:17.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11438 cs=0 l=1 rev1=1 rx=0x7f21e40a6530 tx=0x7f21e4008e60).stop 2021-03-09T10:24:17.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:17.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:17.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:17.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:17.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:17.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ced0 con 0x7f21cc005e30 2021-03-09T10:24:17.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c680 con 0x7f21c4003cb0 2021-03-09T10:24:17.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041b90 con 0x7f21cc042f20 2021-03-09T10:24:17.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:17.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:17.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:17.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:17.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:17.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:17.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:17.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:17.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:17.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c9c0 con 0x7f21cc042f20 2021-03-09T10:24:17.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017800 con 0x7f21cc042f20 2021-03-09T10:24:17.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.176+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b50 secure :-1 s=READY pgs=11450 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc005c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:17.185 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:17.180+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39146 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:17.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:24:17.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081020 con 0x7f21cc042f20 2021-03-09T10:24:17.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc042f20 2021-03-09T10:24:17.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.188+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:17.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.188+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:17.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:17.188+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:22.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:22.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:22.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b50 secure :-1 s=READY pgs=11450 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc005c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:22.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b50 secure :-1 s=READY pgs=11450 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc005c00).stop 2021-03-09T10:24:22.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:22.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b50 unknown :-1 s=CLOSED pgs=11450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:22.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:22.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:22.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:22.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ced0 con 0x7f21cc005e30 2021-03-09T10:24:22.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d4b0 con 0x7f21c4003cb0 2021-03-09T10:24:22.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c680 con 0x7f21cc0138f0 2021-03-09T10:24:22.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:22.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:22.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:22.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:22.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:22.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:22.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:22.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:22.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:22.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0419f0 con 0x7f21cc0138f0 2021-03-09T10:24:22.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.184+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:24:22.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.188+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11464 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d630).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:22.193 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:22.188+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39181 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:22.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d1d0 con 0x7f21cc0138f0 2021-03-09T10:24:22.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003dd0 con 0x7f21cc0138f0 2021-03-09T10:24:22.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006670 con 0x7f21cc0138f0 2021-03-09T10:24:22.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.192+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:22.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:22.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:22.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:26.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:26.194 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:27.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:27.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:27.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11464 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d630).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:27.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11464 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d630).stop 2021-03-09T10:24:27.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:27.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:27.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:27.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:27.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:27.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043e40 con 0x7f21cc005e30 2021-03-09T10:24:27.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d4b0 con 0x7f21c4003cb0 2021-03-09T10:24:27.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:27.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026b70 con 0x7f21cc042f20 2021-03-09T10:24:27.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:27.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:27.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:27.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:27.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:27.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:27.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:27.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:27.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016ff0 con 0x7f21cc042f20 2021-03-09T10:24:27.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02d010 con 0x7f21cc042f20 2021-03-09T10:24:27.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.195+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b00 secure :-1 s=READY pgs=11476 cs=0 l=1 rev1=1 rx=0x7f21e404f7b0 tx=0x7f21e4091190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:27.206 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:27.195+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39221 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:27.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.195+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016ab0 con 0x7f21cc042f20 2021-03-09T10:24:27.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400d050 con 0x7f21cc042f20 2021-03-09T10:24:27.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:24:27.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.203+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:27.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.203+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:27.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:27.203+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:32.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:32.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:32.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b00 secure :-1 s=READY pgs=11476 cs=0 l=1 rev1=1 rx=0x7f21e404f7b0 tx=0x7f21e4091190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:32.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b00 secure :-1 s=READY pgs=11476 cs=0 l=1 rev1=1 rx=0x7f21e404f7b0 tx=0x7f21e4091190).stop 2021-03-09T10:24:32.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:32.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016b00 unknown :-1 s=CLOSED pgs=11476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:32.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:32.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:32.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:32.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:32.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f60 con 0x7f21cc005e30 2021-03-09T10:24:32.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:32.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d4b0 con 0x7f21c4003cb0 2021-03-09T10:24:32.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043e40 con 0x7f21cc0138f0 2021-03-09T10:24:32.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:32.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:32.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:32.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:32.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:32.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:32.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:32.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c680 con 0x7f21cc0138f0 2021-03-09T10:24:32.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c9c0 con 0x7f21cc0138f0 2021-03-09T10:24:32.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.199+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11480 cs=0 l=1 rev1=1 rx=0x7f21cc004ef0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:32.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc0138f0 2021-03-09T10:24:32.210 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:32.203+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39256 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:32.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050c20 con 0x7f21cc0138f0 2021-03-09T10:24:32.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b3d00 con 0x7f21cc0138f0 2021-03-09T10:24:32.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.207+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:32.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.207+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:32.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:32.207+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:37.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:37.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:37.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11480 cs=0 l=1 rev1=1 rx=0x7f21cc004ef0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:37.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11480 cs=0 l=1 rev1=1 rx=0x7f21cc004ef0 tx=0x7f21dc077de0).stop 2021-03-09T10:24:37.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:37.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:37.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:37.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:37.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:37.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043e40 con 0x7f21cc005e30 2021-03-09T10:24:37.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041d20 con 0x7f21c4003cb0 2021-03-09T10:24:37.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f10 con 0x7f21cc042f20 2021-03-09T10:24:37.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:37.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:37.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:37.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:37.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.207+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:37.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:37.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:37.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:37.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:37.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016e20 con 0x7f21cc042f20 2021-03-09T10:24:37.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0419f0 con 0x7f21cc042f20 2021-03-09T10:24:37.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016930 secure :-1 s=READY pgs=11491 cs=0 l=1 rev1=1 rx=0x7f21e0098f20 tx=0x7f21e0075bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:37.216 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:37.215+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39296 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:37.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a0000 con 0x7f21cc042f20 2021-03-09T10:24:37.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a0000 con 0x7f21cc042f20 2021-03-09T10:24:37.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:24:37.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.223+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:37.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.223+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:37.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:37.223+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:42.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:42.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:42.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016930 secure :-1 s=READY pgs=11491 cs=0 l=1 rev1=1 rx=0x7f21e0098f20 tx=0x7f21e0075bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:42.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016930 secure :-1 s=READY pgs=11491 cs=0 l=1 rev1=1 rx=0x7f21e0098f20 tx=0x7f21e0075bb0).stop 2021-03-09T10:24:42.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:42.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016930 unknown :-1 s=CLOSED pgs=11491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:42.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:42.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:42.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:42.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:42.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043e40 con 0x7f21cc005e30 2021-03-09T10:24:42.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc00 con 0x7f21c4003cb0 2021-03-09T10:24:42.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:42.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f10 con 0x7f21cc0138f0 2021-03-09T10:24:42.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:42.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:42.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.215+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:42.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:42.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:42.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:42.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:42.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b1d0 con 0x7f21cc0138f0 2021-03-09T10:24:42.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016ff0 con 0x7f21cc0138f0 2021-03-09T10:24:42.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.219+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11502 cs=0 l=1 rev1=1 rx=0x7f21e403dda0 tx=0x7f21e407fb30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:42.224 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:42.223+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39331 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:42.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e403de10 con 0x7f21cc0138f0 2021-03-09T10:24:42.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.227+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403db70 con 0x7f21cc0138f0 2021-03-09T10:24:42.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:24:42.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.231+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:42.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.231+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:42.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:42.231+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:47.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:47.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:47.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11502 cs=0 l=1 rev1=1 rx=0x7f21e403dda0 tx=0x7f21e407fb30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:47.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=11502 cs=0 l=1 rev1=1 rx=0x7f21e403dda0 tx=0x7f21e407fb30).stop 2021-03-09T10:24:47.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:47.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=11502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:47.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:47.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:47.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:47.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043e40 con 0x7f21cc005e30 2021-03-09T10:24:47.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017190 con 0x7f21c4003cb0 2021-03-09T10:24:47.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f10 con 0x7f21cc042f20 2021-03-09T10:24:47.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:47.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:47.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:47.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:47.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.223+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:47.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:47.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:47.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:47.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:47.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016c60 con 0x7f21cc042f20 2021-03-09T10:24:47.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016e20 con 0x7f21cc042f20 2021-03-09T10:24:47.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.227+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016770 secure :-1 s=READY pgs=11513 cs=0 l=1 rev1=1 rx=0x7f21e007e130 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:47.248 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:47.231+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39371 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:47.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:24:47.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00136c0 con 0x7f21cc042f20 2021-03-09T10:24:47.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:24:47.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.239+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:47.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.239+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:47.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:47.239+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:24:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:51.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 0x7f21cc049d80 secure :-1 s=READY pgs=9114 cs=0 l=1 rev1=1 rx=0x7f21cc030660 tx=0x7f21dc056ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 0x7f21cc049d80 secure :-1 s=READY pgs=9114 cs=0 l=1 rev1=1 rx=0x7f21cc030660 tx=0x7f21dc056ec0).stop 2021-03-09T10:24:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:24:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21cc047360 2021-03-09T10:24:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T10:24:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 msgr2=0x7f21cc049d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc047360 0x7f21cc049d80 unknown :-1 s=CLOSED pgs=9114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:51.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=10923 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc056ec0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:52.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:52.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:52.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016770 secure :-1 s=READY pgs=11513 cs=0 l=1 rev1=1 rx=0x7f21e007e130 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:52.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016770 secure :-1 s=READY pgs=11513 cs=0 l=1 rev1=1 rx=0x7f21e007e130 tx=0x7f21e000b3e0).stop 2021-03-09T10:24:52.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:52.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016770 unknown :-1 s=CLOSED pgs=11513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:52.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc049d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:52.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:52.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc039b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:52.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c9d0 con 0x7f21cc005e30 2021-03-09T10:24:52.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f10 con 0x7f21c4003cb0 2021-03-09T10:24:52.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016c60 con 0x7f21cc047360 2021-03-09T10:24:52.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:52.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:52.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc049d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:52.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc049d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:52.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc039b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:52.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 msgr2=0x7f21cc039b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:24:52.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc047360 0x7f21cc039b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:52.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc01c190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:52.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc01c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:52.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc049d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:52.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc049d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:52.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21c40037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:52.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21c400a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:52.240 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.231+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c400a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:52.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c9d0 con 0x7f21cc005e30 2021-03-09T10:24:52.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f10 con 0x7f21c4003cb0 2021-03-09T10:24:52.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016c60 con 0x7f21c4005da0 2021-03-09T10:24:52.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21c400a1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:52.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21c400a1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:52.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21c400a710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:52.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c400a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:52.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.235+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21c40037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:52.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21c400a1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:52.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21c400a1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:52.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21c400a710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:52.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21c400a710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:52.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016da0 con 0x7f21c4005da0 2021-03-09T10:24:52.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4005da0 2021-03-09T10:24:52.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21c40037c0 secure :-1 s=READY pgs=11530 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0037560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:52.245 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39406 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:52.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21c4005da0 2021-03-09T10:24:52.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0013ad0 con 0x7f21c4005da0 2021-03-09T10:24:52.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00adda0 con 0x7f21c4005da0 2021-03-09T10:24:52.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.247+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:52.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.247+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:52.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:52.247+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:24:56.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:56.242 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:24:57.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21c40037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:57.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21c40037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:57.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21c40037c0 secure :-1 s=READY pgs=11530 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0037560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:57.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21c40037c0 secure :-1 s=READY pgs=11530 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0037560).stop 2021-03-09T10:24:57.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21c40037c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:57.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21c40037c0 unknown :-1 s=CLOSED pgs=11530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:57.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:57.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:57.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:57.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a2a0 con 0x7f21cc005e30 2021-03-09T10:24:57.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017f10 con 0x7f21c4003cb0 2021-03-09T10:24:57.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a040 con 0x7f21cc042f20 2021-03-09T10:24:57.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:24:57.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:57.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:24:57.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.239+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:57.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.243+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:57.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:57.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:57.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:57.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:57.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016aa0 con 0x7f21cc042f20 2021-03-09T10:24:57.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016da0 con 0x7f21cc042f20 2021-03-09T10:24:57.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 secure :-1 s=READY pgs=11541 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0037f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:57.249 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39446 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:24:57.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:24:57.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21cc042f20 2021-03-09T10:24:57.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:24:57.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.255+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:24:57.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.255+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:24:57.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:24:57.255+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:02.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:25:02.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:02.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 secure :-1 s=READY pgs=11541 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0037f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:02.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 secure :-1 s=READY pgs=11541 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0037f60).stop 2021-03-09T10:25:02.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc047360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:02.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 unknown :-1 s=CLOSED pgs=11541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:02.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc016e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:02.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:02.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:02.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096d60 con 0x7f21cc005e30 2021-03-09T10:25:02.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec50 con 0x7f21c4003cb0 2021-03-09T10:25:02.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21c4005da0 2021-03-09T10:25:02.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:02.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:02.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:02.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.247+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:02.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc016e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:02.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:02.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:02.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:02.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:02.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21c4005da0 2021-03-09T10:25:02.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc016aa0 con 0x7f21c4005da0 2021-03-09T10:25:02.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc016e20 secure :-1 s=READY pgs=11546 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:02.257 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:02.255+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39481 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:02.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0040 con 0x7f21c4005da0 2021-03-09T10:25:02.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0028ad0 con 0x7f21c4005da0 2021-03-09T10:25:02.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4005da0 2021-03-09T10:25:02.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.259+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:02.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.259+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:02.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:02.259+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:07.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc016e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:07.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc016e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:07.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc016e20 secure :-1 s=READY pgs=11546 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:07.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc016e20 secure :-1 s=READY pgs=11546 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d540).stop 2021-03-09T10:25:07.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc016e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:07.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc016e20 unknown :-1 s=CLOSED pgs=11546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:07.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:07.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:07.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:07.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040580 con 0x7f21cc005e30 2021-03-09T10:25:07.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047850 con 0x7f21c4003cb0 2021-03-09T10:25:07.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016aa0 con 0x7f21cc042f20 2021-03-09T10:25:07.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:07.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:07.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:07.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:07.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.255+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:07.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:07.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:07.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:07.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:07.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a100 con 0x7f21cc042f20 2021-03-09T10:25:07.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc042f20 2021-03-09T10:25:07.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.259+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 secure :-1 s=READY pgs=11557 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e0075c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:07.265 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:07.263+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39521 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:07.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:25:07.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0020120 con 0x7f21cc042f20 2021-03-09T10:25:07.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:25:07.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.267+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:07.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.267+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:07.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:07.267+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:12.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:12.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc047360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:12.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 secure :-1 s=READY pgs=11557 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e0075c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:12.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 secure :-1 s=READY pgs=11557 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e0075c20).stop 2021-03-09T10:25:12.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc047360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:12.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc047360 unknown :-1 s=CLOSED pgs=11557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:12.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc017b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:12.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:12.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:12.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040580 con 0x7f21cc005e30 2021-03-09T10:25:12.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016aa0 con 0x7f21c4003cb0 2021-03-09T10:25:12.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21c4005da0 2021-03-09T10:25:12.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:12.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:12.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:12.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:12.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.263+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc017b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:12.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:12.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:12.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:12.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:12.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc016c80 con 0x7f21c4005da0 2021-03-09T10:25:12.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a100 con 0x7f21c4005da0 2021-03-09T10:25:12.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.267+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc017b30 secure :-1 s=READY pgs=11568 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0021180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:12.274 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:12.271+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39556 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:12.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.271+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21c4005da0 2021-03-09T10:25:12.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.279+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e001fe10 con 0x7f21c4005da0 2021-03-09T10:25:12.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.279+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a0950 con 0x7f21c4005da0 2021-03-09T10:25:12.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.279+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:12.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.279+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:12.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:12.279+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:17.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc017b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:17.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc017b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:17.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc017b30 secure :-1 s=READY pgs=11568 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0021180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:17.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc017b30 secure :-1 s=READY pgs=11568 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0021180).stop 2021-03-09T10:25:17.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc017b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:17.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc017b30 unknown :-1 s=CLOSED pgs=11568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:17.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:17.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:17.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:17.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a100 con 0x7f21cc005e30 2021-03-09T10:25:17.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040720 con 0x7f21c4003cb0 2021-03-09T10:25:17.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016f90 con 0x7f21cc042f20 2021-03-09T10:25:17.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:17.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:17.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:17.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:17.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.270+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:17.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:17.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:17.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:17.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:17.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017100 con 0x7f21cc042f20 2021-03-09T10:25:17.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c680 con 0x7f21cc042f20 2021-03-09T10:25:17.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.274+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11576 cs=0 l=1 rev1=1 rx=0x7f21cc046630 tx=0x7f21dc005c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:17.280 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:17.278+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39596 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:17.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.278+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0695d0 con 0x7f21cc042f20 2021-03-09T10:25:17.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.278+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21cc042f20 2021-03-09T10:25:17.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc042f20 2021-03-09T10:25:17.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.282+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:17.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.282+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:17.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:17.282+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:22.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:22.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:22.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11576 cs=0 l=1 rev1=1 rx=0x7f21cc046630 tx=0x7f21dc005c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:22.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11576 cs=0 l=1 rev1=1 rx=0x7f21cc046630 tx=0x7f21dc005c00).stop 2021-03-09T10:25:22.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:22.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=CLOSED pgs=11576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:22.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:22.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:22.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:22.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cb70 con 0x7f21cc005e30 2021-03-09T10:25:22.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21c4003cb0 2021-03-09T10:25:22.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042400 con 0x7f21c4005da0 2021-03-09T10:25:22.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:22.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:22.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:22.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:22.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.278+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:22.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:22.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:22.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:22.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:22.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a100 con 0x7f21c4005da0 2021-03-09T10:25:22.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084040 con 0x7f21c4005da0 2021-03-09T10:25:22.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.282+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11587 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:22.292 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:22.286+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39631 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:22.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00837b0 con 0x7f21c4005da0 2021-03-09T10:25:22.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00835d0 con 0x7f21c4005da0 2021-03-09T10:25:22.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.290+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21c4005da0 2021-03-09T10:25:22.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.290+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:22.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.290+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:22.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:22.290+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:26.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:26.292 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:27.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:25:27.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:27.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11587 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:27.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11587 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003740).stop 2021-03-09T10:25:27.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:27.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=11587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:27.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:27.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:27.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:27.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cb70 con 0x7f21cc005e30 2021-03-09T10:25:27.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21c4003cb0 2021-03-09T10:25:27.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016f90 con 0x7f21cc042f20 2021-03-09T10:25:27.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:27.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:27.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:27.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:27.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.286+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:27.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:27.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:27.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:27.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:27.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0408c0 con 0x7f21cc042f20 2021-03-09T10:25:27.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017100 con 0x7f21cc042f20 2021-03-09T10:25:27.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.290+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11598 cs=0 l=1 rev1=1 rx=0x7f21cc027970 tx=0x7f21dc062f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:27.297 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:27.294+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39671 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:27.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00a760 con 0x7f21cc042f20 2021-03-09T10:25:27.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.298+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00a760 con 0x7f21cc042f20 2021-03-09T10:25:27.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.298+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:25:27.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.298+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:27.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.298+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:27.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:27.298+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:32.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:32.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:32.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11598 cs=0 l=1 rev1=1 rx=0x7f21cc027970 tx=0x7f21dc062f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:32.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11598 cs=0 l=1 rev1=1 rx=0x7f21cc027970 tx=0x7f21dc062f70).stop 2021-03-09T10:25:32.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:32.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=CLOSED pgs=11598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:32.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:32.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:32.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:32.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cb70 con 0x7f21cc005e30 2021-03-09T10:25:32.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f960 con 0x7f21c4003cb0 2021-03-09T10:25:32.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a460 con 0x7f21c4005da0 2021-03-09T10:25:32.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:32.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:32.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:32.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:32.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.294+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:32.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:32.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:32.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:32.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:32.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00cc20 con 0x7f21c4005da0 2021-03-09T10:25:32.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a100 con 0x7f21c4005da0 2021-03-09T10:25:32.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.298+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11608 cs=0 l=1 rev1=1 rx=0x7f21e00068d0 tx=0x7f21e002aee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:32.305 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:32.302+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39706 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:32.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.306+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0021e20 con 0x7f21c4005da0 2021-03-09T10:25:32.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21c4005da0 2021-03-09T10:25:32.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4005da0 2021-03-09T10:25:32.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.310+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:32.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.310+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:32.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:32.310+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:37.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:37.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:37.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11608 cs=0 l=1 rev1=1 rx=0x7f21e00068d0 tx=0x7f21e002aee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:37.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11608 cs=0 l=1 rev1=1 rx=0x7f21e00068d0 tx=0x7f21e002aee0).stop 2021-03-09T10:25:37.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:37.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=11608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:37.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:37.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:37.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:37.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a460 con 0x7f21cc005e30 2021-03-09T10:25:37.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d480 con 0x7f21c4003cb0 2021-03-09T10:25:37.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21cc042f20 2021-03-09T10:25:37.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:37.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:37.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:37.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:37.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:37.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.306+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:37.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.306+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:37.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.306+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:37.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.306+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:37.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.306+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a100 con 0x7f21cc042f20 2021-03-09T10:25:37.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.306+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b1d0 con 0x7f21cc042f20 2021-03-09T10:25:37.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.310+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11622 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e40074b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:37.313 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:37.310+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39746 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:37.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T10:25:37.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc042f20 2021-03-09T10:25:37.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f2e0 con 0x7f21cc042f20 2021-03-09T10:25:37.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.318+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:37.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.318+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:37.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:37.318+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:42.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:42.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:42.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11622 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e40074b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:42.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 secure :-1 s=READY pgs=11622 cs=0 l=1 rev1=1 rx=0x7f21e401b060 tx=0x7f21e40074b0).stop 2021-03-09T10:25:42.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:42.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016aa0 unknown :-1 s=CLOSED pgs=11622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:42.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:42.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:42.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:42.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21cc005e30 2021-03-09T10:25:42.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021980 con 0x7f21c4003cb0 2021-03-09T10:25:42.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b370 con 0x7f21c4005da0 2021-03-09T10:25:42.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:42.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:42.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:42.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:42.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.310+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:42.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:42.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:42.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:42.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:42.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc015ce0 con 0x7f21c4005da0 2021-03-09T10:25:42.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a100 con 0x7f21c4005da0 2021-03-09T10:25:42.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.314+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d510 secure :-1 s=READY pgs=11630 cs=0 l=1 rev1=1 rx=0x7f21e407eb50 tx=0x7f21e407e800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:42.320 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:42.318+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39781 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:42.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4005da0 2021-03-09T10:25:42.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1370 con 0x7f21c4005da0 2021-03-09T10:25:42.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.322+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4005da0 2021-03-09T10:25:42.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.322+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:42.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.326+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:42.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:42.326+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:47.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:25:47.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:47.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d510 secure :-1 s=READY pgs=11630 cs=0 l=1 rev1=1 rx=0x7f21e407eb50 tx=0x7f21e407e800).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:47.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d510 secure :-1 s=READY pgs=11630 cs=0 l=1 rev1=1 rx=0x7f21e407eb50 tx=0x7f21e407e800).stop 2021-03-09T10:25:47.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:47.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d510 unknown :-1 s=CLOSED pgs=11630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:47.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:47.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:47.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:47.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928d0 con 0x7f21cc005e30 2021-03-09T10:25:47.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21c4003cb0 2021-03-09T10:25:47.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0222e0 con 0x7f21cc042f20 2021-03-09T10:25:47.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:47.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:47.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:47.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:47.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.318+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:47.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:47.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:47.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:47.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:47.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b370 con 0x7f21cc042f20 2021-03-09T10:25:47.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00cc20 con 0x7f21cc042f20 2021-03-09T10:25:47.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016c40 secure :-1 s=READY pgs=11640 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e007d5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:47.329 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:47.326+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39820 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:47.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc042f20 2021-03-09T10:25:47.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc042f20 2021-03-09T10:25:47.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:25:47.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.330+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:47.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:47.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:47.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:52.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:52.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:52.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016c40 secure :-1 s=READY pgs=11640 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e007d5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:52.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016c40 secure :-1 s=READY pgs=11640 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e007d5b0).stop 2021-03-09T10:25:52.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc016c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:52.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc016c40 unknown :-1 s=CLOSED pgs=11640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:52.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:52.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:52.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:52.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21cc005e30 2021-03-09T10:25:52.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928d0 con 0x7f21c4003cb0 2021-03-09T10:25:52.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036a10 con 0x7f21c4005da0 2021-03-09T10:25:52.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:52.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:52.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:52.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:52.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.326+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:52.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:52.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:52.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:52.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:52.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00cc20 con 0x7f21c4005da0 2021-03-09T10:25:52.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0408c0 con 0x7f21c4005da0 2021-03-09T10:25:52.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11651 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc032200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:52.412 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:52.334+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39856 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:52.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:25:52.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:25:52.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4005da0 2021-03-09T10:25:52.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.342+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:52.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.342+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:52.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:52.342+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:25:56.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:56.344 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:25:57.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:25:57.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:57.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11651 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc032200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:57.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11651 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc032200).stop 2021-03-09T10:25:57.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:57.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:57.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:57.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:57.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:57.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029f30 con 0x7f21cc005e30 2021-03-09T10:25:57.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf20 con 0x7f21c4003cb0 2021-03-09T10:25:57.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21cc042f20 2021-03-09T10:25:57.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:25:57.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:25:57.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:57.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:57.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:57.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:57.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:57.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:57.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:57.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0408c0 con 0x7f21cc042f20 2021-03-09T10:25:57.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a100 con 0x7f21cc042f20 2021-03-09T10:25:57.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=11662 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e40240e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:57.345 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:25:57.342+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39895 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:25:57.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5da0 con 0x7f21cc042f20 2021-03-09T10:25:57.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc042f20 2021-03-09T10:25:57.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5da0 con 0x7f21cc042f20 2021-03-09T10:25:57.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.354+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:25:57.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:25:57.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:25:57.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:02.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:02.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:02.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=11662 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e40240e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:02.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=11662 cs=0 l=1 rev1=1 rx=0x7f21e4003c30 tx=0x7f21e40240e0).stop 2021-03-09T10:26:02.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:02.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=11662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:02.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:02.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:02.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:02.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f40 con 0x7f21cc005e30 2021-03-09T10:26:02.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21c4003cb0 2021-03-09T10:26:02.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096e80 con 0x7f21c4005da0 2021-03-09T10:26:02.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:02.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:02.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:02.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:02.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:02.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:02.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:02.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:02.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:02.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017150 con 0x7f21c4005da0 2021-03-09T10:26:02.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b370 con 0x7f21c4005da0 2021-03-09T10:26:02.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11674 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e00078f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:02.354 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:02.350+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39931 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:02.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:26:02.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:26:02.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:26:02.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.358+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:02.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:02.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:02.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:07.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:07.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:07.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11674 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e00078f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:07.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11674 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e00078f0).stop 2021-03-09T10:26:07.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:07.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:07.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:07.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:07.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:07.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21cc005e30 2021-03-09T10:26:07.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f20 con 0x7f21c4003cb0 2021-03-09T10:26:07.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096e60 con 0x7f21cc042f20 2021-03-09T10:26:07.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:07.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:07.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:07.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:07.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:07.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:07.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:07.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:07.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:07.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b370 con 0x7f21cc042f20 2021-03-09T10:26:07.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00cc20 con 0x7f21cc042f20 2021-03-09T10:26:07.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.353+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=11683 cs=0 l=1 rev1=1 rx=0x7f21cc047900 tx=0x7f21dc0800b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:07.362 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:07.357+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39970 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:07.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.357+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:26:07.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.357+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:26:07.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc042f20 2021-03-09T10:26:07.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.361+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:07.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.365+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:07.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:07.365+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:12.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:12.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:12.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=11683 cs=0 l=1 rev1=1 rx=0x7f21cc047900 tx=0x7f21dc0800b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:12.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=11683 cs=0 l=1 rev1=1 rx=0x7f21cc047900 tx=0x7f21dc0800b0).stop 2021-03-09T10:26:12.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:12.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=11683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:12.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:12.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:12.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:12.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928d0 con 0x7f21cc005e30 2021-03-09T10:26:12.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030ef0 con 0x7f21c4003cb0 2021-03-09T10:26:12.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc20 con 0x7f21c4005da0 2021-03-09T10:26:12.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:12.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:12.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:12.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:12.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:12.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:12.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:12.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:12.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:12.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21c4005da0 2021-03-09T10:26:12.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017150 con 0x7f21c4005da0 2021-03-09T10:26:12.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.361+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11695 cs=0 l=1 rev1=1 rx=0x7f21e0016010 tx=0x7f21e00076f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:12.371 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:12.365+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40006 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:12.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21c4005da0 2021-03-09T10:26:12.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.373+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00768b0 con 0x7f21c4005da0 2021-03-09T10:26:12.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:26:12.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.377+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:12.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.377+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:12.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:12.377+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:17.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:17.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:17.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11695 cs=0 l=1 rev1=1 rx=0x7f21e0016010 tx=0x7f21e00076f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:17.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11695 cs=0 l=1 rev1=1 rx=0x7f21e0016010 tx=0x7f21e00076f0).stop 2021-03-09T10:26:17.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:17.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:17.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:17.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:17.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:17.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053720 con 0x7f21cc005e30 2021-03-09T10:26:17.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0174c0 con 0x7f21c4003cb0 2021-03-09T10:26:17.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc20 con 0x7f21cc042f20 2021-03-09T10:26:17.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:17.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:17.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:17.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:17.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.369+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:17.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:17.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:17.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:17.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:17.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057540 con 0x7f21cc042f20 2021-03-09T10:26:17.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0408c0 con 0x7f21cc042f20 2021-03-09T10:26:17.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f21cc047930 tx=0x7f21e408f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:17.379 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:17.373+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40045 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:17.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc042f20 2021-03-09T10:26:17.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40092e0 con 0x7f21cc042f20 2021-03-09T10:26:17.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.381+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc042f20 2021-03-09T10:26:17.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.381+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:17.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.381+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:17.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:17.381+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:22.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:22.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:22.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f21cc047930 tx=0x7f21e408f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:22.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f21cc047930 tx=0x7f21e408f220).stop 2021-03-09T10:26:22.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:22.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:22.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:22.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:22.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:22.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0408c0 con 0x7f21cc005e30 2021-03-09T10:26:22.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cc20 con 0x7f21c4003cb0 2021-03-09T10:26:22.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21c4005da0 2021-03-09T10:26:22.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:22.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:22.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:22.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:22.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:22.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:22.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:22.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:22.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:22.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04dca0 con 0x7f21c4005da0 2021-03-09T10:26:22.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.377+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b370 con 0x7f21c4005da0 2021-03-09T10:26:22.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.381+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11715 cs=0 l=1 rev1=1 rx=0x7f21cc00ca50 tx=0x7f21dc018fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:22.393 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:22.385+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40081 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:22.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:26:22.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:26:22.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc068a40 con 0x7f21c4005da0 2021-03-09T10:26:22.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.389+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:22.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.389+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:22.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:22.389+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:26.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:26.393 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:27.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:27.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:27.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11715 cs=0 l=1 rev1=1 rx=0x7f21cc00ca50 tx=0x7f21dc018fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:27.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11715 cs=0 l=1 rev1=1 rx=0x7f21cc00ca50 tx=0x7f21dc018fa0).stop 2021-03-09T10:26:27.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:27.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:27.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:27.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:27.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:27.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21cc005e30 2021-03-09T10:26:27.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b370 con 0x7f21c4003cb0 2021-03-09T10:26:27.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0408c0 con 0x7f21cc042f20 2021-03-09T10:26:27.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:27.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:27.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:27.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:27.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:27.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:27.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:27.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:27.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:27.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00cc20 con 0x7f21cc042f20 2021-03-09T10:26:27.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057540 con 0x7f21cc042f20 2021-03-09T10:26:27.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11727 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e40b1f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:27.396 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:27.393+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40119 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:27.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc042f20 2021-03-09T10:26:27.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016240 con 0x7f21cc042f20 2021-03-09T10:26:27.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc042f20 2021-03-09T10:26:27.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.397+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:27.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:27.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:27.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:32.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:32.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:32.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11727 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e40b1f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:32.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11727 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e40b1f80).stop 2021-03-09T10:26:32.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:32.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:32.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:32.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:32.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:32.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21cc005e30 2021-03-09T10:26:32.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dfd0 con 0x7f21c4003cb0 2021-03-09T10:26:32.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a470 con 0x7f21c4005da0 2021-03-09T10:26:32.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:32.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:32.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:32.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:32.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:32.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:32.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:32.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:32.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:32.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b370 con 0x7f21c4005da0 2021-03-09T10:26:32.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.393+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21c4005da0 2021-03-09T10:26:32.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.397+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11736 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e007d5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:32.404 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:32.401+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40155 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:32.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:26:32.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:26:32.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:26:32.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.405+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:32.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.405+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:32.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:32.405+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:37.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:37.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:37.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11736 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e007d5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:37.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11736 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e007d5b0).stop 2021-03-09T10:26:37.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:37.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:37.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:37.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:37.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:37.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21cc005e30 2021-03-09T10:26:37.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:37.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04dfd0 con 0x7f21c4003cb0 2021-03-09T10:26:37.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cdd0 con 0x7f21cc042f20 2021-03-09T10:26:37.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:37.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:37.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:37.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.401+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:37.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:37.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:37.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:37.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:37.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0408c0 con 0x7f21cc042f20 2021-03-09T10:26:37.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00cc20 con 0x7f21cc042f20 2021-03-09T10:26:37.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11750 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e407f6b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:37.412 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:37.409+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40194 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:37.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5d40 con 0x7f21cc042f20 2021-03-09T10:26:37.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4003a50 con 0x7f21cc042f20 2021-03-09T10:26:37.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5d40 con 0x7f21cc042f20 2021-03-09T10:26:37.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.413+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:37.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:37.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:37.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:42.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:42.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:42.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11750 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e407f6b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:42.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11750 cs=0 l=1 rev1=1 rx=0x7f21e4037040 tx=0x7f21e407f6b0).stop 2021-03-09T10:26:42.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:42.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:42.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:42.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:42.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:42.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ce0 con 0x7f21cc005e30 2021-03-09T10:26:42.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043be0 con 0x7f21c4003cb0 2021-03-09T10:26:42.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08dfc0 con 0x7f21c4005da0 2021-03-09T10:26:42.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:42.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:42.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:42.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:42.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:42.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:42.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:42.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:42.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:42.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00cc20 con 0x7f21c4005da0 2021-03-09T10:26:42.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.409+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04dca0 con 0x7f21c4005da0 2021-03-09T10:26:42.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11758 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc015a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:42.420 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:42.413+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40230 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:42.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c4005da0 2021-03-09T10:26:42.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21c4005da0 2021-03-09T10:26:42.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c4005da0 2021-03-09T10:26:42.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.421+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:42.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:42.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:42.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:47.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:26:47.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:47.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11758 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc015a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:47.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11758 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc015a90).stop 2021-03-09T10:26:47.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:47.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:47.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:47.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:47.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:47.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:47.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e4b0 con 0x7f21cc005e30 2021-03-09T10:26:47.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:47.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04de40 con 0x7f21c4003cb0 2021-03-09T10:26:47.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438a0 con 0x7f21cc042f20 2021-03-09T10:26:47.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:47.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:47.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:47.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:47.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:47.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:47.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:47.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b370 con 0x7f21cc042f20 2021-03-09T10:26:47.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00cc20 con 0x7f21cc042f20 2021-03-09T10:26:47.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.421+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11771 cs=0 l=1 rev1=1 rx=0x7f21dc015bd0 tx=0x7f21dc0285f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:47.427 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:47.425+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40268 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:47.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc028000 con 0x7f21cc042f20 2021-03-09T10:26:47.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028000 con 0x7f21cc042f20 2021-03-09T10:26:47.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc028d00 con 0x7f21cc042f20 2021-03-09T10:26:47.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.429+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:47.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.433+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:47.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:47.433+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:52.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:52.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:52.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11771 cs=0 l=1 rev1=1 rx=0x7f21dc015bd0 tx=0x7f21dc0285f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:52.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11771 cs=0 l=1 rev1=1 rx=0x7f21dc015bd0 tx=0x7f21dc0285f0).stop 2021-03-09T10:26:52.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:52.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:52.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:52.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:52.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:52.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c420 con 0x7f21cc005e30 2021-03-09T10:26:52.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e4b0 con 0x7f21c4003cb0 2021-03-09T10:26:52.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030640 con 0x7f21c4005da0 2021-03-09T10:26:52.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:52.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:52.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:52.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:52.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:52.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:52.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:52.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:52.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:52.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21c4005da0 2021-03-09T10:26:52.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b370 con 0x7f21c4005da0 2021-03-09T10:26:52.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.429+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11780 cs=0 l=1 rev1=1 rx=0x7f21dc028580 tx=0x7f21dc08a1a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:52.435 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:52.429+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40305 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:52.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07eb10 con 0x7f21c4005da0 2021-03-09T10:26:52.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07eb10 con 0x7f21c4005da0 2021-03-09T10:26:52.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc028d00 con 0x7f21c4005da0 2021-03-09T10:26:52.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.437+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:52.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.437+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:52.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:52.437+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:26:56.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:56.441 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:26:57.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:26:57.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:57.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11780 cs=0 l=1 rev1=1 rx=0x7f21dc028580 tx=0x7f21dc08a1a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:57.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11780 cs=0 l=1 rev1=1 rx=0x7f21dc028580 tx=0x7f21dc08a1a0).stop 2021-03-09T10:26:57.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:57.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:57.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:57.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:57.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:57.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e4b0 con 0x7f21cc005e30 2021-03-09T10:26:57.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21c4003cb0 2021-03-09T10:26:57.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030d70 con 0x7f21cc042f20 2021-03-09T10:26:57.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:26:57.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:57.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:26:57.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:57.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:57.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:57.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:57.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:57.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:57.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b370 con 0x7f21cc042f20 2021-03-09T10:26:57.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:26:57.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.432+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11790 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0721a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:57.444 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:26:57.440+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40342 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:26:57.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc042f20 2021-03-09T10:26:57.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc042f20 2021-03-09T10:26:57.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:26:57.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.440+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:26:57.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.444+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:26:57.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:26:57.444+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:02.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:27:02.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:02.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11790 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0721a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:02.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11790 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0721a0).stop 2021-03-09T10:27:02.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:02.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:02.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:02.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:02.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:02.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:02.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e4b0 con 0x7f21cc005e30 2021-03-09T10:27:02.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:02.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f10 con 0x7f21c4003cb0 2021-03-09T10:27:02.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21c4005da0 2021-03-09T10:27:02.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:02.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:02.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.440+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:02.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:02.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:02.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:02.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:02.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04de40 con 0x7f21c4005da0 2021-03-09T10:27:02.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b370 con 0x7f21c4005da0 2021-03-09T10:27:02.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.444+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11802 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:02.452 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:02.448+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:02.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.448+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4005da0 2021-03-09T10:27:02.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4005da0 2021-03-09T10:27:02.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4005da0 2021-03-09T10:27:02.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.456+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:02.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.456+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:02.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:02.456+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:07.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:07.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:07.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11802 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:07.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11802 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc010d20).stop 2021-03-09T10:27:07.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:07.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:07.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:07.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:07.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:07.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008160 con 0x7f21cc005e30 2021-03-09T10:27:07.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f10 con 0x7f21c4003cb0 2021-03-09T10:27:07.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b370 con 0x7f21cc042f20 2021-03-09T10:27:07.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:07.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:07.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:07.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:07.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:07.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:07.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:07.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:07.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:07.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:27:07.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04de40 con 0x7f21cc042f20 2021-03-09T10:27:07.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.452+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11811 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc083450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:07.460 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:07.456+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40417 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:07.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005e10 con 0x7f21cc042f20 2021-03-09T10:27:07.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00a130 con 0x7f21cc042f20 2021-03-09T10:27:07.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.460+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21cc042f20 2021-03-09T10:27:07.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.460+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:07.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.460+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:07.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:07.460+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:12.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:27:12.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:12.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11811 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc083450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:12.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11811 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc083450).stop 2021-03-09T10:27:12.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:12.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:12.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:12.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:12.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:12.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc005e30 2021-03-09T10:27:12.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f10 con 0x7f21c4003cb0 2021-03-09T10:27:12.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008140 con 0x7f21c4005da0 2021-03-09T10:27:12.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:12.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:12.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:12.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:12.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:12.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:12.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:12.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:12.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:12.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d9b0 con 0x7f21c4005da0 2021-03-09T10:27:12.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.456+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21c4005da0 2021-03-09T10:27:12.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.460+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11822 cs=0 l=1 rev1=1 rx=0x7f21e002e030 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:12.468 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:12.464+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40455 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:12.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:27:12.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.464+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21c4005da0 2021-03-09T10:27:12.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:27:12.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.468+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:12.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.468+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:12.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:12.468+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:17.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:17.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:17.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11822 cs=0 l=1 rev1=1 rx=0x7f21e002e030 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:17.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 secure :-1 s=READY pgs=11822 cs=0 l=1 rev1=1 rx=0x7f21e002e030 tx=0x7f21e0003820).stop 2021-03-09T10:27:17.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:17.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d4c0 unknown :-1 s=CLOSED pgs=11822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:17.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:17.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:17.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:17.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc005e30 2021-03-09T10:27:17.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f10 con 0x7f21c4003cb0 2021-03-09T10:27:17.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e180 con 0x7f21cc042f20 2021-03-09T10:27:17.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:17.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:17.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:17.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:17.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:17.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:17.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:17.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:17.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:17.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d9b0 con 0x7f21cc042f20 2021-03-09T10:27:17.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:27:17.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.468+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11832 cs=0 l=1 rev1=1 rx=0x7f21cc04a610 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:17.477 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:17.472+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40492 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:17.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.472+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00c3e0 con 0x7f21cc042f20 2021-03-09T10:27:17.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00c1b0 con 0x7f21cc042f20 2021-03-09T10:27:17.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21cc042f20 2021-03-09T10:27:17.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.480+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:17.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:17.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:17.480+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:22.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:22.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:22.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11832 cs=0 l=1 rev1=1 rx=0x7f21cc04a610 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:22.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11832 cs=0 l=1 rev1=1 rx=0x7f21cc04a610 tx=0x7f21dc02dfc0).stop 2021-03-09T10:27:22.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:22.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:22.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:22.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:22.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:22.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030430 con 0x7f21cc005e30 2021-03-09T10:27:22.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21c4003cb0 2021-03-09T10:27:22.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e180 con 0x7f21c4005da0 2021-03-09T10:27:22.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:22.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:22.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:22.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:22.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:22.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:22.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:22.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:22.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:22.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d4c0 con 0x7f21c4005da0 2021-03-09T10:27:22.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0408c0 con 0x7f21c4005da0 2021-03-09T10:27:22.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11843 cs=0 l=1 rev1=1 rx=0x7f21e4018000 tx=0x7f21e4027ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:22.487 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:22.480+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40530 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:22.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:27:22.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:27:22.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40081c0 con 0x7f21c4005da0 2021-03-09T10:27:22.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.488+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:22.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:22.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:22.492+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:26.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:26.489 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:27.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:27.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:27.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11843 cs=0 l=1 rev1=1 rx=0x7f21e4018000 tx=0x7f21e4027ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:27.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11843 cs=0 l=1 rev1=1 rx=0x7f21e4018000 tx=0x7f21e4027ed0).stop 2021-03-09T10:27:27.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:27.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=CLOSED pgs=11843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:27.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.480+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:27.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:27.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:27.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc005e30 2021-03-09T10:27:27.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094be0 con 0x7f21c4003cb0 2021-03-09T10:27:27.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0408c0 con 0x7f21cc042f20 2021-03-09T10:27:27.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:27.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:27.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:27.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:27.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:27.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:27.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:27.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:27.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:27.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d650 con 0x7f21cc042f20 2021-03-09T10:27:27.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d4c0 con 0x7f21cc042f20 2021-03-09T10:27:27.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11854 cs=0 l=1 rev1=1 rx=0x7f21cc04e0c0 tx=0x7f21e4009440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:27.494 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:27.488+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40567 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:27.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409cc80 con 0x7f21cc042f20 2021-03-09T10:27:27.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5e10 con 0x7f21cc042f20 2021-03-09T10:27:27.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cc80 con 0x7f21cc042f20 2021-03-09T10:27:27.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.496+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:27.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.496+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:27.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:27.496+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:32.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:32.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:32.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11854 cs=0 l=1 rev1=1 rx=0x7f21cc04e0c0 tx=0x7f21e4009440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:32.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 secure :-1 s=READY pgs=11854 cs=0 l=1 rev1=1 rx=0x7f21cc04e0c0 tx=0x7f21e4009440).stop 2021-03-09T10:27:32.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc01cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:32.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc01cb70 unknown :-1 s=CLOSED pgs=11854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:32.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:32.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:32.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:32.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:32.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:32.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc005e30 2021-03-09T10:27:32.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d4c0 con 0x7f21c4003cb0 2021-03-09T10:27:32.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d7e0 con 0x7f21c4005da0 2021-03-09T10:27:32.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:32.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:32.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:32.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:32.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:32.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:32.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:32.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d920 con 0x7f21c4005da0 2021-03-09T10:27:32.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d650 con 0x7f21c4005da0 2021-03-09T10:27:32.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.492+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11864 cs=0 l=1 rev1=1 rx=0x7f21e404f2f0 tx=0x7f21e400f2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:32.502 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:32.496+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40605 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:32.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402b3e0 con 0x7f21c4005da0 2021-03-09T10:27:32.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402b200 con 0x7f21c4005da0 2021-03-09T10:27:32.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028120 con 0x7f21c4005da0 2021-03-09T10:27:32.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.504+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:32.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.504+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:32.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:32.504+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:37.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:37.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:37.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11864 cs=0 l=1 rev1=1 rx=0x7f21e404f2f0 tx=0x7f21e400f2f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:37.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11864 cs=0 l=1 rev1=1 rx=0x7f21e404f2f0 tx=0x7f21e400f2f0).stop 2021-03-09T10:27:37.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:37.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=CLOSED pgs=11864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:37.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:37.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:37.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:37.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0265a0 con 0x7f21cc005e30 2021-03-09T10:27:37.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053840 con 0x7f21c4003cb0 2021-03-09T10:27:37.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc042f20 2021-03-09T10:27:37.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:37.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:37.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:37.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:37.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:37.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:37.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:37.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:37.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:37.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030780 con 0x7f21cc042f20 2021-03-09T10:27:37.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21cc042f20 2021-03-09T10:27:37.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.500+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11875 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc00d4a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:37.511 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:37.508+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40642 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:37.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00d220 con 0x7f21cc042f20 2021-03-09T10:27:37.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc042f20 2021-03-09T10:27:37.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:27:37.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.508+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:37.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.512+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:37.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:37.512+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:42.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:42.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:42.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11875 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc00d4a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:42.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11875 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc00d4a0).stop 2021-03-09T10:27:42.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:42.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=CLOSED pgs=11875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:42.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:42.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:42.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:42.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d80 con 0x7f21cc005e30 2021-03-09T10:27:42.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21c4003cb0 2021-03-09T10:27:42.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026740 con 0x7f21c4005da0 2021-03-09T10:27:42.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:42.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:42.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:42.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:42.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:42.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:42.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:42.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:42.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:42.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21c4005da0 2021-03-09T10:27:42.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030780 con 0x7f21c4005da0 2021-03-09T10:27:42.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11887 cs=0 l=1 rev1=1 rx=0x7f21cc092940 tx=0x7f21dc0052b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:42.520 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:42.516+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:42.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.516+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc028810 con 0x7f21c4005da0 2021-03-09T10:27:42.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc028810 con 0x7f21c4005da0 2021-03-09T10:27:42.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4005da0 2021-03-09T10:27:42.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.524+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:42.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.524+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:42.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:42.524+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:47.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:47.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:47.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11887 cs=0 l=1 rev1=1 rx=0x7f21cc092940 tx=0x7f21dc0052b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:47.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11887 cs=0 l=1 rev1=1 rx=0x7f21cc092940 tx=0x7f21dc0052b0).stop 2021-03-09T10:27:47.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:47.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=CLOSED pgs=11887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:47.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:47.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:47.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:47.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047650 con 0x7f21cc005e30 2021-03-09T10:27:47.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c910 con 0x7f21c4003cb0 2021-03-09T10:27:47.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d80 con 0x7f21cc042f20 2021-03-09T10:27:47.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:47.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:47.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:47.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:47.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:47.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:47.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:47.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:47.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:47.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030780 con 0x7f21cc042f20 2021-03-09T10:27:47.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc042f20 2021-03-09T10:27:47.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.519+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11896 cs=0 l=1 rev1=1 rx=0x7f21e005b010 tx=0x7f21e00a6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:47.532 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:47.523+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40717 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:47.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0005f40 con 0x7f21cc042f20 2021-03-09T10:27:47.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.523+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006170 con 0x7f21cc042f20 2021-03-09T10:27:47.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:27:47.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.527+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:47.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.527+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:47.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:47.527+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:52.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:52.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:52.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11896 cs=0 l=1 rev1=1 rx=0x7f21e005b010 tx=0x7f21e00a6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:52.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11896 cs=0 l=1 rev1=1 rx=0x7f21e005b010 tx=0x7f21e00a6200).stop 2021-03-09T10:27:52.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:52.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=CLOSED pgs=11896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:52.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:52.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:52.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:52.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc005e30 2021-03-09T10:27:52.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d780 con 0x7f21c4003cb0 2021-03-09T10:27:52.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:27:52.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:52.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:52.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:52.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:52.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.527+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:52.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:52.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:52.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:52.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:52.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c910 con 0x7f21c4005da0 2021-03-09T10:27:52.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d920 con 0x7f21c4005da0 2021-03-09T10:27:52.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.531+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11911 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e40b43e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:52.536 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:52.535+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40754 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:52.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21c4005da0 2021-03-09T10:27:52.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.539+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4003a70 con 0x7f21c4005da0 2021-03-09T10:27:52.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.539+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21c4005da0 2021-03-09T10:27:52.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.539+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:52.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.543+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:52.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:52.543+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:27:56.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:56.536 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:27:57.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:57.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:57.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11911 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e40b43e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:57.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 secure :-1 s=READY pgs=11911 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e40b43e0).stop 2021-03-09T10:27:57.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01b090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:57.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01b090 unknown :-1 s=CLOSED pgs=11911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:57.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:57.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:57.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:57.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0438e0 con 0x7f21cc005e30 2021-03-09T10:27:57.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21c4003cb0 2021-03-09T10:27:57.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d490 con 0x7f21cc042f20 2021-03-09T10:27:57.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:27:57.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:57.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:27:57.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:57.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.535+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:57.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:57.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:57.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:57.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:57.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d5d0 con 0x7f21cc042f20 2021-03-09T10:27:57.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:27:57.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.539+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11918 cs=0 l=1 rev1=1 rx=0x7f21dc0330a0 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:57.545 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:27:57.543+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40792 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:27:57.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.547+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc042f20 2021-03-09T10:27:57.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc042f20 2021-03-09T10:27:57.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:27:57.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.551+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:27:57.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.551+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:27:57.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:27:57.551+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:02.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:02.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:02.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11918 cs=0 l=1 rev1=1 rx=0x7f21dc0330a0 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:02.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11918 cs=0 l=1 rev1=1 rx=0x7f21dc0330a0 tx=0x7f21dc009290).stop 2021-03-09T10:28:02.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:02.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=CLOSED pgs=11918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:02.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:02.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:02.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:02.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:02.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:02.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d490 con 0x7f21cc005e30 2021-03-09T10:28:02.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21c4003cb0 2021-03-09T10:28:02.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b090 con 0x7f21c4005da0 2021-03-09T10:28:02.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:02.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:02.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.543+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:02.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:02.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:02.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:02.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:02.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b200 con 0x7f21c4005da0 2021-03-09T10:28:02.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c910 con 0x7f21c4005da0 2021-03-09T10:28:02.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.547+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d760 secure :-1 s=READY pgs=11928 cs=0 l=1 rev1=1 rx=0x7f21e40b02a0 tx=0x7f21e4028350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:02.553 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:02.551+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40829 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:02.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4059000 con 0x7f21c4005da0 2021-03-09T10:28:02.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4059000 con 0x7f21c4005da0 2021-03-09T10:28:02.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4005da0 2021-03-09T10:28:02.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.559+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:02.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.559+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:02.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:02.559+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:07.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:07.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:07.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d760 secure :-1 s=READY pgs=11928 cs=0 l=1 rev1=1 rx=0x7f21e40b02a0 tx=0x7f21e4028350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:07.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d760 secure :-1 s=READY pgs=11928 cs=0 l=1 rev1=1 rx=0x7f21e40b02a0 tx=0x7f21e4028350).stop 2021-03-09T10:28:07.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc08d760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:07.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc08d760 unknown :-1 s=CLOSED pgs=11928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:07.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:07.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:07.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:07.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b090 con 0x7f21cc005e30 2021-03-09T10:28:07.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c910 con 0x7f21c4003cb0 2021-03-09T10:28:07.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:28:07.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:07.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:07.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:07.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.551+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:07.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:07.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:07.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:07.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:07.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:07.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:28:07.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d5d0 con 0x7f21cc042f20 2021-03-09T10:28:07.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.555+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11940 cs=0 l=1 rev1=1 rx=0x7f21dc069ca0 tx=0x7f21dc032ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:07.562 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:07.559+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40867 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:07.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc042f20 2021-03-09T10:28:07.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc042f20 2021-03-09T10:28:07.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc042f20 2021-03-09T10:28:07.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.567+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:07.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.567+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:07.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:07.567+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:12.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:12.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:12.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11940 cs=0 l=1 rev1=1 rx=0x7f21dc069ca0 tx=0x7f21dc032ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:12.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11940 cs=0 l=1 rev1=1 rx=0x7f21dc069ca0 tx=0x7f21dc032ff0).stop 2021-03-09T10:28:12.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:12.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=CLOSED pgs=11940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:12.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:12.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:12.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:12.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.559+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b090 con 0x7f21cc005e30 2021-03-09T10:28:12.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097530 con 0x7f21c4003cb0 2021-03-09T10:28:12.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035dd0 con 0x7f21c4005da0 2021-03-09T10:28:12.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:12.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:12.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:12.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:12.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:12.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:12.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:12.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:12.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:12.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c910 con 0x7f21c4005da0 2021-03-09T10:28:12.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:28:12.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.563+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11950 cs=0 l=1 rev1=1 rx=0x7f21dc0b32e0 tx=0x7f21dc0b3350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:12.571 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:12.567+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40904 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:12.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.571+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4005da0 2021-03-09T10:28:12.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4005da0 2021-03-09T10:28:12.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21c4005da0 2021-03-09T10:28:12.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.575+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:12.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.575+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:12.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:12.575+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:17.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:28:17.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:17.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11950 cs=0 l=1 rev1=1 rx=0x7f21dc0b32e0 tx=0x7f21dc0b3350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:17.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11950 cs=0 l=1 rev1=1 rx=0x7f21dc0b32e0 tx=0x7f21dc0b3350).stop 2021-03-09T10:28:17.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:17.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=11950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:17.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:17.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:17.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:17.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b090 con 0x7f21cc005e30 2021-03-09T10:28:17.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035dd0 con 0x7f21c4003cb0 2021-03-09T10:28:17.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e160 con 0x7f21cc042f20 2021-03-09T10:28:17.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.567+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:17.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:17.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:17.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:17.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:17.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:17.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:17.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:17.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:17.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:28:17.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.571+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030780 con 0x7f21cc042f20 2021-03-09T10:28:17.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11961 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:17.579 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:17.575+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40942 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:17.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc042f20 2021-03-09T10:28:17.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0059880 con 0x7f21cc042f20 2021-03-09T10:28:17.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:28:17.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.579+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:17.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.579+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:17.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:17.579+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:22.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:22.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:22.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11961 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:22.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11961 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e000b3e0).stop 2021-03-09T10:28:22.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:22.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=CLOSED pgs=11961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:22.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:22.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:22.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:22.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:22.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035dd0 con 0x7f21cc005e30 2021-03-09T10:28:22.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b090 con 0x7f21c4003cb0 2021-03-09T10:28:22.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094750 con 0x7f21c4005da0 2021-03-09T10:28:22.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:22.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:22.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:22.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:22.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:22.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:22.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:22.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:22.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095400 con 0x7f21c4005da0 2021-03-09T10:28:22.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b200 con 0x7f21c4005da0 2021-03-09T10:28:22.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.579+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11971 cs=0 l=1 rev1=1 rx=0x7f21cc042880 tx=0x7f21e4028120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:22.590 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:22.583+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40980 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:22.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003a60 con 0x7f21c4005da0 2021-03-09T10:28:22.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4003a60 con 0x7f21c4005da0 2021-03-09T10:28:22.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.587+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:28:22.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.587+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:22.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.587+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:22.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:22.587+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:26.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:26.652 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:27.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:27.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:27.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11971 cs=0 l=1 rev1=1 rx=0x7f21cc042880 tx=0x7f21e4028120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:27.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=11971 cs=0 l=1 rev1=1 rx=0x7f21cc042880 tx=0x7f21e4028120).stop 2021-03-09T10:28:27.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:27.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=11971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:27.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:27.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:27.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:27.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02efd0 con 0x7f21cc005e30 2021-03-09T10:28:27.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041d00 con 0x7f21c4003cb0 2021-03-09T10:28:27.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b3a0 con 0x7f21cc042f20 2021-03-09T10:28:27.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:27.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:27.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:27.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.583+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:27.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.587+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:27.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:27.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:27.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:27.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:27.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c2d0 con 0x7f21cc042f20 2021-03-09T10:28:27.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:28:27.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11987 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e007d580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:27.594 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:27.591+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41017 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:27.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0081470 con 0x7f21cc042f20 2021-03-09T10:28:27.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0081470 con 0x7f21cc042f20 2021-03-09T10:28:27.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:28:27.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.599+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:27.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:27.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:27.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:32.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:28:32.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:32.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11987 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e007d580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:32.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 secure :-1 s=READY pgs=11987 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e007d580).stop 2021-03-09T10:28:32.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc030290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:32.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc030290 unknown :-1 s=CLOSED pgs=11987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:32.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:32.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc089500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:32.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:32.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:32.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21cc005e30 2021-03-09T10:28:32.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:32.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02efd0 con 0x7f21c4003cb0 2021-03-09T10:28:32.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d090 con 0x7f21c4005da0 2021-03-09T10:28:32.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:32.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:32.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc089500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:32.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:32.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:32.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:32.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:32.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d200 con 0x7f21c4005da0 2021-03-09T10:28:32.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095400 con 0x7f21c4005da0 2021-03-09T10:28:32.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc089500 secure :-1 s=READY pgs=11992 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:32.603 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:32.599+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41055 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:32.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4005da0 2021-03-09T10:28:32.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a68f0 con 0x7f21c4005da0 2021-03-09T10:28:32.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4005da0 2021-03-09T10:28:32.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.603+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:32.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:32.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:32.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:37.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc089500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:37.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc089500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:37.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc089500 secure :-1 s=READY pgs=11992 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:37.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc089500 secure :-1 s=READY pgs=11992 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4003ad0).stop 2021-03-09T10:28:37.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc089500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:37.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc089500 unknown :-1 s=CLOSED pgs=11992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:37.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:37.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:37.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:37.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095400 con 0x7f21cc005e30 2021-03-09T10:28:37.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21c4003cb0 2021-03-09T10:28:37.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.598+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:28:37.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:37.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:37.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:37.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:37.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:37.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:37.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:37.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:37.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:37.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d040 con 0x7f21cc042f20 2021-03-09T10:28:37.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.602+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c910 con 0x7f21cc042f20 2021-03-09T10:28:37.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.606+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12004 cs=0 l=1 rev1=1 rx=0x7f21cc047800 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:37.610 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:37.606+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41092 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:37.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc042f20 2021-03-09T10:28:37.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc042f20 2021-03-09T10:28:37.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc021180 con 0x7f21cc042f20 2021-03-09T10:28:37.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.614+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:37.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.614+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:37.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:37.614+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:42.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:42.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:42.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12004 cs=0 l=1 rev1=1 rx=0x7f21cc047800 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:42.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12004 cs=0 l=1 rev1=1 rx=0x7f21cc047800 tx=0x7f21dc02df50).stop 2021-03-09T10:28:42.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:42.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=12004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:42.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:42.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:42.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:42.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ae0 con 0x7f21cc005e30 2021-03-09T10:28:42.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4003cb0 2021-03-09T10:28:42.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:28:42.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:42.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:42.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:42.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:42.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:42.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:42.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:42.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:42.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:42.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c910 con 0x7f21c4005da0 2021-03-09T10:28:42.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.610+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d040 con 0x7f21c4005da0 2021-03-09T10:28:42.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084fe0 secure :-1 s=READY pgs=12016 cs=0 l=1 rev1=1 rx=0x7f21cc0429c0 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:42.619 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:42.614+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41130 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:42.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052380 con 0x7f21c4005da0 2021-03-09T10:28:42.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21c4005da0 2021-03-09T10:28:42.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.622+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052380 con 0x7f21c4005da0 2021-03-09T10:28:42.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.622+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:42.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.626+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:42.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:42.626+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:47.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:47.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:47.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084fe0 secure :-1 s=READY pgs=12016 cs=0 l=1 rev1=1 rx=0x7f21cc0429c0 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:47.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084fe0 secure :-1 s=READY pgs=12016 cs=0 l=1 rev1=1 rx=0x7f21cc0429c0 tx=0x7f21dc010d90).stop 2021-03-09T10:28:47.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:47.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084fe0 unknown :-1 s=CLOSED pgs=12016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:47.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.614+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:47.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:47.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:47.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ae0 con 0x7f21cc005e30 2021-03-09T10:28:47.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095060 con 0x7f21c4003cb0 2021-03-09T10:28:47.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e60 con 0x7f21cc042f20 2021-03-09T10:28:47.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:47.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:47.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:47.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:47.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:47.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:47.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:47.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:47.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:47.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095400 con 0x7f21cc042f20 2021-03-09T10:28:47.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c910 con 0x7f21cc042f20 2021-03-09T10:28:47.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.618+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12025 cs=0 l=1 rev1=1 rx=0x7f21cc02eee0 tx=0x7f21dc0811f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:47.629 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:47.622+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:47.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07e7f0 con 0x7f21cc042f20 2021-03-09T10:28:47.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07e7f0 con 0x7f21cc042f20 2021-03-09T10:28:47.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.630+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc052310 con 0x7f21cc042f20 2021-03-09T10:28:47.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.630+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:47.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:47.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:47.630+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:52.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:52.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:52.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12025 cs=0 l=1 rev1=1 rx=0x7f21cc02eee0 tx=0x7f21dc0811f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:52.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12025 cs=0 l=1 rev1=1 rx=0x7f21cc02eee0 tx=0x7f21dc0811f0).stop 2021-03-09T10:28:52.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:52.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=12025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:52.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:52.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:52.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:52.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:52.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ae0 con 0x7f21cc005e30 2021-03-09T10:28:52.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:52.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e60 con 0x7f21c4003cb0 2021-03-09T10:28:52.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1a0 con 0x7f21c4005da0 2021-03-09T10:28:52.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:52.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:52.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:52.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:52.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:52.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:52.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:52.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:28:52.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.626+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095400 con 0x7f21c4005da0 2021-03-09T10:28:52.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.630+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088e80 secure :-1 s=READY pgs=12036 cs=0 l=1 rev1=1 rx=0x7f21dc032fc0 tx=0x7f21dc087f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:52.635 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:52.634+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:52.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc07e7f0 con 0x7f21c4005da0 2021-03-09T10:28:52.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc07e7f0 con 0x7f21c4005da0 2021-03-09T10:28:52.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.638+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4005da0 2021-03-09T10:28:52.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.638+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:52.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.638+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:52.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:52.638+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:28:56.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:56.714 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:28:57.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc088e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:57.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc088e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:57.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088e80 secure :-1 s=READY pgs=12036 cs=0 l=1 rev1=1 rx=0x7f21dc032fc0 tx=0x7f21dc087f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:57.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088e80 secure :-1 s=READY pgs=12036 cs=0 l=1 rev1=1 rx=0x7f21dc032fc0 tx=0x7f21dc087f90).stop 2021-03-09T10:28:57.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc088e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:57.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088e80 unknown :-1 s=CLOSED pgs=12036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:57.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:57.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:57.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:28:57.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:57.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:57.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e60 con 0x7f21cc005e30 2021-03-09T10:28:57.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d1a0 con 0x7f21c4003cb0 2021-03-09T10:28:57.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21cc042f20 2021-03-09T10:28:57.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:28:57.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:57.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:57.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:57.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:57.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036f00 con 0x7f21cc042f20 2021-03-09T10:28:57.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.634+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d200 con 0x7f21cc042f20 2021-03-09T10:28:57.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.638+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12046 cs=0 l=1 rev1=1 rx=0x7f21e4016240 tx=0x7f21e4016350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:57.643 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:28:57.642+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41241 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:28:57.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4041350 con 0x7f21cc042f20 2021-03-09T10:28:57.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4041350 con 0x7f21cc042f20 2021-03-09T10:28:57.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.646+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407fd30 con 0x7f21cc042f20 2021-03-09T10:28:57.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.646+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:28:57.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.650+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:28:57.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:28:57.650+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:02.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:02.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:02.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12046 cs=0 l=1 rev1=1 rx=0x7f21e4016240 tx=0x7f21e4016350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:02.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12046 cs=0 l=1 rev1=1 rx=0x7f21e4016240 tx=0x7f21e4016350).stop 2021-03-09T10:29:02.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:02.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=12046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:02.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:02.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:02.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:02.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc004e60 con 0x7f21cc005e30 2021-03-09T10:29:02.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4003cb0 2021-03-09T10:29:02.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21c4005da0 2021-03-09T10:29:02.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:02.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:02.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:02.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:02.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:02.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:02.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:02.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:02.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:02.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095400 con 0x7f21c4005da0 2021-03-09T10:29:02.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.642+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c2d0 con 0x7f21c4005da0 2021-03-09T10:29:02.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.646+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088b40 secure :-1 s=READY pgs=12056 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e00037c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:02.652 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:02.650+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41280 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:02.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0081880 con 0x7f21c4005da0 2021-03-09T10:29:02.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0081880 con 0x7f21c4005da0 2021-03-09T10:29:02.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.654+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:29:02.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.654+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:02.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.654+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:02.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:02.654+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:07.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc088b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:07.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc088b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:07.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088b40 secure :-1 s=READY pgs=12056 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e00037c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:07.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088b40 secure :-1 s=READY pgs=12056 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e00037c0).stop 2021-03-09T10:29:07.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc088b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:07.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc088b40 unknown :-1 s=CLOSED pgs=12056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:07.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc089030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:07.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:07.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:07.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21cc005e30 2021-03-09T10:29:07.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21c4003cb0 2021-03-09T10:29:07.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21cc042f20 2021-03-09T10:29:07.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:07.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:07.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:07.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:07.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc089030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:07.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:07.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:07.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:07.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:07.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c2d0 con 0x7f21cc042f20 2021-03-09T10:29:07.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.650+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095400 con 0x7f21cc042f20 2021-03-09T10:29:07.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.654+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc089030 secure :-1 s=READY pgs=12067 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0075cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:07.661 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:07.658+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41316 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:07.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0081700 con 0x7f21cc042f20 2021-03-09T10:29:07.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0081860 con 0x7f21cc042f20 2021-03-09T10:29:07.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.662+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc042f20 2021-03-09T10:29:07.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.662+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:07.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.662+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:07.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:07.662+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:12.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc089030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:12.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc089030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:12.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc089030 secure :-1 s=READY pgs=12067 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0075cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:12.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc089030 secure :-1 s=READY pgs=12067 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0075cb0).stop 2021-03-09T10:29:12.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc089030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:12.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc089030 unknown :-1 s=CLOSED pgs=12067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:12.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:12.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:12.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:12.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21cc005e30 2021-03-09T10:29:12.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21c4003cb0 2021-03-09T10:29:12.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053a00 con 0x7f21c4005da0 2021-03-09T10:29:12.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:12.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:12.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:12.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:12.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.658+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:12.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:12.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:12.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:12.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:12.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c910 con 0x7f21c4005da0 2021-03-09T10:29:12.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:29:12.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.662+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12078 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc031de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:12.675 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:12.666+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41355 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:12.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0501f0 con 0x7f21c4005da0 2021-03-09T10:29:12.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0503c0 con 0x7f21c4005da0 2021-03-09T10:29:12.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.670+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4005da0 2021-03-09T10:29:12.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.670+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:12.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.670+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:12.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:12.670+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:17.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:17.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:17.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12078 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc031de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:17.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12078 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc031de0).stop 2021-03-09T10:29:17.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:17.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:17.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:17.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:17.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:17.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:17.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:17.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21cc005e30 2021-03-09T10:29:17.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4003cb0 2021-03-09T10:29:17.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096f20 con 0x7f21cc042f20 2021-03-09T10:29:17.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:17.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:17.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.666+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:17.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:17.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:17.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:17.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:17.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:29:17.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c910 con 0x7f21cc042f20 2021-03-09T10:29:17.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.670+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088e90 secure :-1 s=READY pgs=12088 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc081000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:17.679 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:17.674+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41392 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:17.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc052f10 con 0x7f21cc042f20 2021-03-09T10:29:17.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc042f20 2021-03-09T10:29:17.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.678+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010800 con 0x7f21cc042f20 2021-03-09T10:29:17.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.678+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:17.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.678+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:17.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:17.678+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:22.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:22.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:22.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088e90 secure :-1 s=READY pgs=12088 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc081000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:22.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088e90 secure :-1 s=READY pgs=12088 cs=0 l=1 rev1=1 rx=0x7f21dc0278c0 tx=0x7f21dc081000).stop 2021-03-09T10:29:22.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:22.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088e90 unknown :-1 s=CLOSED pgs=12088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:22.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:22.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:22.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:22.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21cc005e30 2021-03-09T10:29:22.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4003cb0 2021-03-09T10:29:22.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094c50 con 0x7f21c4005da0 2021-03-09T10:29:22.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:22.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:22.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:22.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:22.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:22.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.678+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:22.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.678+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:22.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.678+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:22.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.678+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:22.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.678+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095400 con 0x7f21c4005da0 2021-03-09T10:29:22.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.678+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:29:22.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.682+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12105 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0183f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:22.685 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:22.682+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41430 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:22.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.686+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0185b0 con 0x7f21c4005da0 2021-03-09T10:29:22.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.686+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c4005da0 2021-03-09T10:29:22.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c4005da0 2021-03-09T10:29:22.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.690+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:22.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.690+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:22.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:22.690+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:26.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:26.769 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:27.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:27.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:27.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12105 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0183f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:27.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12105 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0183f0).stop 2021-03-09T10:29:27.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:27.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:27.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:27.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:27.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:27.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:27.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:27.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21cc005e30 2021-03-09T10:29:27.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4003cb0 2021-03-09T10:29:27.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.681+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cda0 con 0x7f21cc042f20 2021-03-09T10:29:27.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:27.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:27.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:27.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:27.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:27.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:27.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:27.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c910 con 0x7f21cc042f20 2021-03-09T10:29:27.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.685+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036f00 con 0x7f21cc042f20 2021-03-09T10:29:27.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.689+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12112 cs=0 l=1 rev1=1 rx=0x7f21cc046110 tx=0x7f21e4036c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:27.694 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:27.689+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41467 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:27.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.689+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016080 con 0x7f21cc042f20 2021-03-09T10:29:27.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc042f20 2021-03-09T10:29:27.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.697+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016080 con 0x7f21cc042f20 2021-03-09T10:29:27.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.697+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:27.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.697+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:27.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:27.697+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:32.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:32.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:32.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12112 cs=0 l=1 rev1=1 rx=0x7f21cc046110 tx=0x7f21e4036c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:32.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12112 cs=0 l=1 rev1=1 rx=0x7f21cc046110 tx=0x7f21e4036c00).stop 2021-03-09T10:29:32.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:32.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=12112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:32.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:32.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:32.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:32.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d180 con 0x7f21cc005e30 2021-03-09T10:29:32.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026f50 con 0x7f21c4003cb0 2021-03-09T10:29:32.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4005da0 2021-03-09T10:29:32.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:32.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:32.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:32.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:32.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:32.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:32.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:32.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:32.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:32.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:29:32.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095400 con 0x7f21c4005da0 2021-03-09T10:29:32.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.693+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12123 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc080040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:32.701 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:32.697+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41505 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:32.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.697+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ac3c0 con 0x7f21c4005da0 2021-03-09T10:29:32.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.697+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ac1e0 con 0x7f21c4005da0 2021-03-09T10:29:32.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.705+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4005da0 2021-03-09T10:29:32.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.705+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:32.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.705+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:32.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:32.705+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:37.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:37.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:37.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12123 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc080040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:37.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 secure :-1 s=READY pgs=12123 cs=0 l=1 rev1=1 rx=0x7f21dc07ccf0 tx=0x7f21dc080040).stop 2021-03-09T10:29:37.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:37.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:37.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:37.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:37.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:37.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:37.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:37.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:37.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:37.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:37.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cda0 con 0x7f21cc005e30 2021-03-09T10:29:37.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c4003cb0 2021-03-09T10:29:37.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02ee20 con 0x7f21cc042f20 2021-03-09T10:29:37.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:37.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:37.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:37.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:37.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:29:37.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c2d0 con 0x7f21cc042f20 2021-03-09T10:29:37.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.701+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12131 cs=0 l=1 rev1=1 rx=0x7f21e00597b0 tx=0x7f21e0091ba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:37.710 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:37.705+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41542 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:37.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.705+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000b3e0 con 0x7f21cc042f20 2021-03-09T10:29:37.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.709+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c590 con 0x7f21cc042f20 2021-03-09T10:29:37.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.709+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:29:37.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.709+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:37.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.709+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:37.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:37.709+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:42.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:42.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:42.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12131 cs=0 l=1 rev1=1 rx=0x7f21e00597b0 tx=0x7f21e0091ba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:42.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12131 cs=0 l=1 rev1=1 rx=0x7f21e00597b0 tx=0x7f21e0091ba0).stop 2021-03-09T10:29:42.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:42.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=12131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:42.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:42.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:42.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:42.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cda0 con 0x7f21cc005e30 2021-03-09T10:29:42.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094c50 con 0x7f21c4003cb0 2021-03-09T10:29:42.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:29:42.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:42.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:42.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:42.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:42.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:42.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:42.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:42.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:42.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:42.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d370 con 0x7f21c4005da0 2021-03-09T10:29:42.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.709+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:29:42.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.713+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 secure :-1 s=READY pgs=12142 cs=0 l=1 rev1=1 rx=0x7f21dc0b7010 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:42.720 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:42.717+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41580 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:42.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.717+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06d9d0 con 0x7f21c4005da0 2021-03-09T10:29:42.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.717+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06dbe0 con 0x7f21c4005da0 2021-03-09T10:29:42.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.721+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21c4005da0 2021-03-09T10:29:42.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.721+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:42.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.721+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:42.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:42.721+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:47.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:47.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:47.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 secure :-1 s=READY pgs=12142 cs=0 l=1 rev1=1 rx=0x7f21dc0b7010 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:47.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 secure :-1 s=READY pgs=12142 cs=0 l=1 rev1=1 rx=0x7f21dc0b7010 tx=0x7f21dc02df50).stop 2021-03-09T10:29:47.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0416d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:47.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 unknown :-1 s=CLOSED pgs=12142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:47.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:47.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:47.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:47.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b4f0 con 0x7f21cc005e30 2021-03-09T10:29:47.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cda0 con 0x7f21c4003cb0 2021-03-09T10:29:47.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:29:47.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:47.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:47.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:47.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:47.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.717+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:47.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:47.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:47.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:47.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:47.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d370 con 0x7f21cc042f20 2021-03-09T10:29:47.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:29:47.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.721+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12152 cs=0 l=1 rev1=1 rx=0x7f21cc04ed10 tx=0x7f21dc010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:47.730 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:47.725+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41617 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:47.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.725+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0509f0 con 0x7f21cc042f20 2021-03-09T10:29:47.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.725+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050e30 con 0x7f21cc042f20 2021-03-09T10:29:47.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc042f20 2021-03-09T10:29:47.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.729+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:47.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.729+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:47.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:47.729+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:52.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:52.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:52.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12152 cs=0 l=1 rev1=1 rx=0x7f21cc04ed10 tx=0x7f21dc010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:52.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12152 cs=0 l=1 rev1=1 rx=0x7f21cc04ed10 tx=0x7f21dc010e50).stop 2021-03-09T10:29:52.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:52.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:52.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cda0 con 0x7f21cc005e30 2021-03-09T10:29:52.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d370 con 0x7f21c4003cb0 2021-03-09T10:29:52.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041a20 con 0x7f21c4005da0 2021-03-09T10:29:52.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:52.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:52.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:52.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:52.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:52.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:52.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:52.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:52.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:52.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021f80 con 0x7f21c4005da0 2021-03-09T10:29:52.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c2d0 con 0x7f21c4005da0 2021-03-09T10:29:52.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.729+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12163 cs=0 l=1 rev1=1 rx=0x7f21e0080c10 tx=0x7f21e007d620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:52.737 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:52.733+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41655 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:52.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:29:52.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.733+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e009d880 con 0x7f21c4005da0 2021-03-09T10:29:52.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.737+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:29:52.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.737+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:52.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.737+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:52.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:52.737+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:29:56.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:56.819 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:29:57.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:57.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:57.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12163 cs=0 l=1 rev1=1 rx=0x7f21e0080c10 tx=0x7f21e007d620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:57.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12163 cs=0 l=1 rev1=1 rx=0x7f21e0080c10 tx=0x7f21e007d620).stop 2021-03-09T10:29:57.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:57.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=12163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:57.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:57.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:57.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:57.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022110 con 0x7f21cc005e30 2021-03-09T10:29:57.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c2d0 con 0x7f21c4003cb0 2021-03-09T10:29:57.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:29:57.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:29:57.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:57.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:57.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:29:57.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:57.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:57.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:57.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:57.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:57.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc088e90 con 0x7f21cc042f20 2021-03-09T10:29:57.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c910 con 0x7f21cc042f20 2021-03-09T10:29:57.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.737+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12174 cs=0 l=1 rev1=1 rx=0x7f21cc007b80 tx=0x7f21e4003a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:57.746 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:29:57.741+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41691 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:29:57.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.741+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a6b00 con 0x7f21cc042f20 2021-03-09T10:29:57.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.741+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc042f20 2021-03-09T10:29:57.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.749+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a6b00 con 0x7f21cc042f20 2021-03-09T10:29:57.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.749+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:29:57.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.749+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:29:57.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:29:57.749+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:02.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:02.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:02.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12174 cs=0 l=1 rev1=1 rx=0x7f21cc007b80 tx=0x7f21e4003a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:02.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12174 cs=0 l=1 rev1=1 rx=0x7f21cc007b80 tx=0x7f21e4003a20).stop 2021-03-09T10:30:02.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:02.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:02.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:02.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:02.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:02.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022110 con 0x7f21cc005e30 2021-03-09T10:30:02.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21c4003cb0 2021-03-09T10:30:02.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041530 con 0x7f21c4005da0 2021-03-09T10:30:02.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:02.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:02.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:02.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:02.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:02.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:02.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:02.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:02.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:02.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041670 con 0x7f21c4005da0 2021-03-09T10:30:02.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c910 con 0x7f21c4005da0 2021-03-09T10:30:02.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.745+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094220 secure :-1 s=READY pgs=12186 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40281e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:02.756 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:02.749+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41730 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:02.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.749+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a3410 con 0x7f21c4005da0 2021-03-09T10:30:02.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40282e0 con 0x7f21c4005da0 2021-03-09T10:30:02.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a3410 con 0x7f21c4005da0 2021-03-09T10:30:02.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.757+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:02.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.757+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:02.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:02.757+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:07.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc094220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:07.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc094220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:07.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094220 secure :-1 s=READY pgs=12186 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40281e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:07.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094220 secure :-1 s=READY pgs=12186 cs=0 l=1 rev1=1 rx=0x7f21e4054060 tx=0x7f21e40281e0).stop 2021-03-09T10:30:07.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc094220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:07.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094220 unknown :-1 s=CLOSED pgs=12186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:07.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:07.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:07.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:07.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:07.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:07.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041530 con 0x7f21cc005e30 2021-03-09T10:30:07.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T10:30:07.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc042f20 2021-03-09T10:30:07.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:07.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:07.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:07.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:07.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:07.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:07.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:07.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c770 con 0x7f21cc042f20 2021-03-09T10:30:07.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021f80 con 0x7f21cc042f20 2021-03-09T10:30:07.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12196 cs=0 l=1 rev1=1 rx=0x7f21e0089010 tx=0x7f21e00a6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:07.763 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:07.757+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41765 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:07.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006780 con 0x7f21cc042f20 2021-03-09T10:30:07.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006780 con 0x7f21cc042f20 2021-03-09T10:30:07.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:30:07.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.765+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:07.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.769+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:07.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:07.769+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:12.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:12.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12196 cs=0 l=1 rev1=1 rx=0x7f21e0089010 tx=0x7f21e00a6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12196 cs=0 l=1 rev1=1 rx=0x7f21e0089010 tx=0x7f21e00a6190).stop 2021-03-09T10:30:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:12.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:12.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d2f0 con 0x7f21cc005e30 2021-03-09T10:30:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d430 con 0x7f21c4003cb0 2021-03-09T10:30:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0943d0 con 0x7f21c4005da0 2021-03-09T10:30:12.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:12.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:12.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:12.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:12.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:12.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:12.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:12.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094510 con 0x7f21c4005da0 2021-03-09T10:30:12.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.761+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:30:12.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093ee0 secure :-1 s=READY pgs=12207 cs=0 l=1 rev1=1 rx=0x7f21cc029f80 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:12.960 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:12.769+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41804 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:30:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:30:12.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4005da0 2021-03-09T10:30:12.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.777+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:12.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:12.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:12.777+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:17.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:17.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:17.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093ee0 secure :-1 s=READY pgs=12207 cs=0 l=1 rev1=1 rx=0x7f21cc029f80 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:17.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093ee0 secure :-1 s=READY pgs=12207 cs=0 l=1 rev1=1 rx=0x7f21cc029f80 tx=0x7f21dc010d90).stop 2021-03-09T10:30:17.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:17.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093ee0 unknown :-1 s=CLOSED pgs=12207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:17.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:17.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:17.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:17.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0943d0 con 0x7f21cc005e30 2021-03-09T10:30:17.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b300 con 0x7f21c4003cb0 2021-03-09T10:30:17.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d470 con 0x7f21cc042f20 2021-03-09T10:30:17.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:17.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:17.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:17.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:17.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:17.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:17.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:17.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:17.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:17.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:30:17.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094510 con 0x7f21cc042f20 2021-03-09T10:30:17.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.768+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12216 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0621e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:17.780 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:17.776+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41840 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:17.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.776+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc072e70 con 0x7f21cc042f20 2021-03-09T10:30:17.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.776+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc072e70 con 0x7f21cc042f20 2021-03-09T10:30:17.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.780+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07ecc0 con 0x7f21cc042f20 2021-03-09T10:30:17.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.780+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:17.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.780+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:17.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:17.780+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:22.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:22.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:22.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12216 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0621e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:22.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12216 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0621e0).stop 2021-03-09T10:30:22.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:22.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:22.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:22.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:22.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:22.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d470 con 0x7f21cc005e30 2021-03-09T10:30:22.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0895c0 con 0x7f21c4003cb0 2021-03-09T10:30:22.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b2b0 con 0x7f21c4005da0 2021-03-09T10:30:22.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:22.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:22.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:22.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:22.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.776+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:22.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:22.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:22.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:22.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:22.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0941e0 con 0x7f21c4005da0 2021-03-09T10:30:22.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c770 con 0x7f21c4005da0 2021-03-09T10:30:22.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.780+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093cf0 secure :-1 s=READY pgs=12227 cs=0 l=1 rev1=1 rx=0x7f21e0088df0 tx=0x7f21e0091b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:22.788 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:22.784+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41879 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:22.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.784+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4005da0 2021-03-09T10:30:22.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.784+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:30:22.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:30:22.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.788+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:22.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.788+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:22.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:22.788+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:26.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:26.866 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:27.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:27.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:27.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093cf0 secure :-1 s=READY pgs=12227 cs=0 l=1 rev1=1 rx=0x7f21e0088df0 tx=0x7f21e0091b60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:27.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093cf0 secure :-1 s=READY pgs=12227 cs=0 l=1 rev1=1 rx=0x7f21e0088df0 tx=0x7f21e0091b60).stop 2021-03-09T10:30:27.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:27.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093cf0 unknown :-1 s=CLOSED pgs=12227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:27.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:27.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:27.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:27.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b2b0 con 0x7f21cc005e30 2021-03-09T10:30:27.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T10:30:27.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc042f20 2021-03-09T10:30:27.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:27.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:27.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:27.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:27.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:27.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:27.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:27.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:27.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:27.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094370 con 0x7f21cc042f20 2021-03-09T10:30:27.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041670 con 0x7f21cc042f20 2021-03-09T10:30:27.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.788+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12238 cs=0 l=1 rev1=1 rx=0x7f21e40a6380 tx=0x7f21e40a63b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:27.798 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:27.792+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41915 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:27.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40412f0 con 0x7f21cc042f20 2021-03-09T10:30:27.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.792+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40412f0 con 0x7f21cc042f20 2021-03-09T10:30:27.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5dd0 con 0x7f21cc042f20 2021-03-09T10:30:27.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.796+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:27.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.796+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:27.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:27.796+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:32.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:32.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:32.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12238 cs=0 l=1 rev1=1 rx=0x7f21e40a6380 tx=0x7f21e40a63b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:32.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12238 cs=0 l=1 rev1=1 rx=0x7f21e40a6380 tx=0x7f21e40a63b0).stop 2021-03-09T10:30:32.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:32.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:32.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:32.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:32.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:32.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041810 con 0x7f21cc005e30 2021-03-09T10:30:32.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c5a0 con 0x7f21c4003cb0 2021-03-09T10:30:32.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c6e0 con 0x7f21c4005da0 2021-03-09T10:30:32.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:32.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:32.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:32.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:32.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:32.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:32.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:32.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:32.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:32.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01c850 con 0x7f21c4005da0 2021-03-09T10:30:32.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:30:32.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.796+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093c80 secure :-1 s=READY pgs=12249 cs=0 l=1 rev1=1 rx=0x7f21cc026dc0 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:32.807 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:32.800+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41954 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:32.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0298b0 con 0x7f21c4005da0 2021-03-09T10:30:32.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0298b0 con 0x7f21c4005da0 2021-03-09T10:30:32.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07ecc0 con 0x7f21c4005da0 2021-03-09T10:30:32.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.808+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:32.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.808+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:32.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:32.808+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:37.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:37.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:37.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093c80 secure :-1 s=READY pgs=12249 cs=0 l=1 rev1=1 rx=0x7f21cc026dc0 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:37.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093c80 secure :-1 s=READY pgs=12249 cs=0 l=1 rev1=1 rx=0x7f21cc026dc0 tx=0x7f21dc02dfc0).stop 2021-03-09T10:30:37.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:37.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093c80 unknown :-1 s=CLOSED pgs=12249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:37.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:37.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:37.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:37.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:37.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b670 con 0x7f21cc005e30 2021-03-09T10:30:37.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:37.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0946a0 con 0x7f21c4003cb0 2021-03-09T10:30:37.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041830 con 0x7f21cc042f20 2021-03-09T10:30:37.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:37.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:37.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:37.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:37.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:37.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:37.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:37.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc036310 con 0x7f21cc042f20 2021-03-09T10:30:37.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01c850 con 0x7f21cc042f20 2021-03-09T10:30:37.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.804+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12260 cs=0 l=1 rev1=1 rx=0x7f21dc007fa0 tx=0x7f21dc007de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:37.813 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:37.808+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41990 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:37.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.808+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:30:37.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc042f20 2021-03-09T10:30:37.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21cc042f20 2021-03-09T10:30:37.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.816+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:37.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.816+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:37.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:37.816+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:42.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:42.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:42.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12260 cs=0 l=1 rev1=1 rx=0x7f21dc007fa0 tx=0x7f21dc007de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:42.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12260 cs=0 l=1 rev1=1 rx=0x7f21dc007fa0 tx=0x7f21dc007de0).stop 2021-03-09T10:30:42.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:42.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:42.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:42.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:42.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:42.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093f80 con 0x7f21cc005e30 2021-03-09T10:30:42.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0308f0 con 0x7f21c4003cb0 2021-03-09T10:30:42.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030a30 con 0x7f21c4005da0 2021-03-09T10:30:42.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:42.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:42.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:42.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:42.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:42.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:42.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:42.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:42.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:42.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01d590 con 0x7f21c4005da0 2021-03-09T10:30:42.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc036310 con 0x7f21c4005da0 2021-03-09T10:30:42.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.812+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093a90 secure :-1 s=READY pgs=12271 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0050e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:42.822 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:42.816+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42029 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:42.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.816+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089830 con 0x7f21c4005da0 2021-03-09T10:30:42.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089830 con 0x7f21c4005da0 2021-03-09T10:30:42.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21c4005da0 2021-03-09T10:30:42.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.824+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:42.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:42.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:42.824+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:47.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:30:47.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:47.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093a90 secure :-1 s=READY pgs=12271 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0050e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:47.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093a90 secure :-1 s=READY pgs=12271 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0050e0).stop 2021-03-09T10:30:47.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc093a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:47.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc093a90 unknown :-1 s=CLOSED pgs=12271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:47.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:47.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:47.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:47.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:47.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036310 con 0x7f21cc005e30 2021-03-09T10:30:47.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:47.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T10:30:47.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc042f20 2021-03-09T10:30:47.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:47.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:47.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:47.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:47.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:47.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:47.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:47.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:30:47.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01d590 con 0x7f21cc042f20 2021-03-09T10:30:47.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.820+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12280 cs=0 l=1 rev1=1 rx=0x7f21cc026dc0 tx=0x7f21dc080040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:47.830 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:47.824+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42065 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:47.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.824+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3300 con 0x7f21cc042f20 2021-03-09T10:30:47.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3120 con 0x7f21cc042f20 2021-03-09T10:30:47.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21cc042f20 2021-03-09T10:30:47.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.828+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:47.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.828+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:47.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:47.828+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:52.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:52.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:52.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12280 cs=0 l=1 rev1=1 rx=0x7f21cc026dc0 tx=0x7f21dc080040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:52.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12280 cs=0 l=1 rev1=1 rx=0x7f21cc026dc0 tx=0x7f21dc080040).stop 2021-03-09T10:30:52.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:52.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:52.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:52.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:52.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:52.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036330 con 0x7f21cc005e30 2021-03-09T10:30:52.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21c4003cb0 2021-03-09T10:30:52.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d770 con 0x7f21c4005da0 2021-03-09T10:30:52.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:52.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:52.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:52.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:52.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.828+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:52.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:52.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:52.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:52.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:52.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093fa0 con 0x7f21c4005da0 2021-03-09T10:30:52.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094370 con 0x7f21c4005da0 2021-03-09T10:30:52.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.832+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084c80 secure :-1 s=READY pgs=12295 cs=0 l=1 rev1=1 rx=0x7f21e400d210 tx=0x7f21e4059130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:52.843 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:52.836+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42104 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:52.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.836+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:30:52.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c4005da0 2021-03-09T10:30:52.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:30:52.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.844+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:52.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:52.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:52.844+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:30:56.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:56.912 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:30:57.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:57.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:57.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084c80 secure :-1 s=READY pgs=12295 cs=0 l=1 rev1=1 rx=0x7f21e400d210 tx=0x7f21e4059130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:57.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084c80 secure :-1 s=READY pgs=12295 cs=0 l=1 rev1=1 rx=0x7f21e400d210 tx=0x7f21e4059130).stop 2021-03-09T10:30:57.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:57.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084c80 unknown :-1 s=CLOSED pgs=12295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:57.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:57.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:57.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:57.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc005e30 2021-03-09T10:30:57.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094510 con 0x7f21c4003cb0 2021-03-09T10:30:57.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:30:57.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:30:57.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:57.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:30:57.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:57.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:57.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:57.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:57.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:57.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:57.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc085170 con 0x7f21cc042f20 2021-03-09T10:30:57.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0941e0 con 0x7f21cc042f20 2021-03-09T10:30:57.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.840+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12303 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:57.849 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:30:57.844+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42140 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:30:57.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21cc042f20 2021-03-09T10:30:57.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.844+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21cc042f20 2021-03-09T10:30:57.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:30:57.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.848+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:30:57.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.848+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:30:57.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:30:57.848+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:02.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:02.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:02.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12303 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:02.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12303 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0003820).stop 2021-03-09T10:31:02.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:02.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:02.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:02.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:02.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:02.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc005e30 2021-03-09T10:31:02.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0576f0 con 0x7f21c4003cb0 2021-03-09T10:31:02.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc085000 con 0x7f21c4005da0 2021-03-09T10:31:02.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:02.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:02.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:02.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:02.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:02.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:02.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:02.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:02.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:02.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094130 con 0x7f21c4005da0 2021-03-09T10:31:02.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc085170 con 0x7f21c4005da0 2021-03-09T10:31:02.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.848+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d3b0 secure :-1 s=READY pgs=12312 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00818f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:02.857 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:02.852+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:02.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21c4005da0 2021-03-09T10:31:02.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.852+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21c4005da0 2021-03-09T10:31:02.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.856+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4005da0 2021-03-09T10:31:02.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.856+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:02.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.856+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:02.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:02.856+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:07.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:07.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:07.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d3b0 secure :-1 s=READY pgs=12312 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00818f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:07.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d3b0 secure :-1 s=READY pgs=12312 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e00818f0).stop 2021-03-09T10:31:07.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:07.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d3b0 unknown :-1 s=CLOSED pgs=12312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:07.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:07.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:07.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:07.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc005e30 2021-03-09T10:31:07.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224c0 con 0x7f21c4003cb0 2021-03-09T10:31:07.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095a40 con 0x7f21cc042f20 2021-03-09T10:31:07.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:07.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:07.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:07.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:07.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:07.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:07.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:07.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.851+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:07.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.855+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:07.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.855+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:31:07.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.855+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc042f20 2021-03-09T10:31:07.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.855+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12323 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0014c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:07.865 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:07.859+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42215 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:07.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21cc042f20 2021-03-09T10:31:07.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.859+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21cc042f20 2021-03-09T10:31:07.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc042f20 2021-03-09T10:31:07.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.863+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:07.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.863+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:07.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:07.863+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:12.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:12.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:12.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12323 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0014c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:12.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12323 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0014c60).stop 2021-03-09T10:31:12.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:12.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:12.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:12.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:12.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:12.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088e90 con 0x7f21cc005e30 2021-03-09T10:31:12.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022330 con 0x7f21c4003cb0 2021-03-09T10:31:12.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01aff0 con 0x7f21c4005da0 2021-03-09T10:31:12.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:12.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:12.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:12.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:12.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:12.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:12.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:12.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:12.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:12.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:31:12.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21c4005da0 2021-03-09T10:31:12.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.863+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 secure :-1 s=READY pgs=12334 cs=0 l=1 rev1=1 rx=0x7f21e0014e20 tx=0x7f21e00067f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:12.874 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:12.867+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42254 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:12.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.867+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21c4005da0 2021-03-09T10:31:12.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e009abc0 con 0x7f21c4005da0 2021-03-09T10:31:12.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4005da0 2021-03-09T10:31:12.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.871+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:12.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.875+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:12.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:12.875+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:17.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.871+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:17.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.871+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:17.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.871+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 secure :-1 s=READY pgs=12334 cs=0 l=1 rev1=1 rx=0x7f21e0014e20 tx=0x7f21e00067f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:17.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.871+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 secure :-1 s=READY pgs=12334 cs=0 l=1 rev1=1 rx=0x7f21e0014e20 tx=0x7f21e00067f0).stop 2021-03-09T10:31:17.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.871+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:17.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 unknown :-1 s=CLOSED pgs=12334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:17.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:17.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:17.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:17.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a90 con 0x7f21cc005e30 2021-03-09T10:31:17.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022de0 con 0x7f21c4003cb0 2021-03-09T10:31:17.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036cc0 con 0x7f21cc042f20 2021-03-09T10:31:17.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:17.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:17.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:17.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:17.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:17.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:17.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:17.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:17.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:17.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:31:17.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc042f20 2021-03-09T10:31:17.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.875+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12344 cs=0 l=1 rev1=1 rx=0x7f21e0083ff0 tx=0x7f21e00817d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:17.883 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:17.883+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42290 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:17.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0081650 con 0x7f21cc042f20 2021-03-09T10:31:17.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0083ac0 con 0x7f21cc042f20 2021-03-09T10:31:17.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21cc042f20 2021-03-09T10:31:17.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.883+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:17.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.887+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:17.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:17.887+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:22.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:22.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:22.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12344 cs=0 l=1 rev1=1 rx=0x7f21e0083ff0 tx=0x7f21e00817d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:22.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 secure :-1 s=READY pgs=12344 cs=0 l=1 rev1=1 rx=0x7f21e0083ff0 tx=0x7f21e00817d0).stop 2021-03-09T10:31:22.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0889a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:22.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0889a0 unknown :-1 s=CLOSED pgs=12344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:22.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:22.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:22.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:22.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0219e0 con 0x7f21cc005e30 2021-03-09T10:31:22.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T10:31:22.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0308d0 con 0x7f21c4005da0 2021-03-09T10:31:22.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:22.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:22.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:22.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:22.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.883+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:22.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:22.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:22.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:22.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:22.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030a10 con 0x7f21c4005da0 2021-03-09T10:31:22.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084af0 con 0x7f21c4005da0 2021-03-09T10:31:22.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.887+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 secure :-1 s=READY pgs=12358 cs=0 l=1 rev1=1 rx=0x7f21dc026fd0 tx=0x7f21dc009340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:22.892 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:22.891+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42329 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:22.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4005da0 2021-03-09T10:31:22.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4005da0 2021-03-09T10:31:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0571d0 con 0x7f21c4005da0 2021-03-09T10:31:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.899+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:22.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.899+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:22.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:22.899+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:26.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:26.954 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:27.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:31:27.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:27.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 secure :-1 s=READY pgs=12358 cs=0 l=1 rev1=1 rx=0x7f21dc026fd0 tx=0x7f21dc009340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:27.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 secure :-1 s=READY pgs=12358 cs=0 l=1 rev1=1 rx=0x7f21dc026fd0 tx=0x7f21dc009340).stop 2021-03-09T10:31:27.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc01d340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:27.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc01d340 unknown :-1 s=CLOSED pgs=12358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:27.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:27.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:27.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:27.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0308d0 con 0x7f21cc005e30 2021-03-09T10:31:27.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016f50 con 0x7f21c4003cb0 2021-03-09T10:31:27.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:27.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021b80 con 0x7f21cc042f20 2021-03-09T10:31:27.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:27.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:27.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:27.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:27.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:27.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:27.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:27.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:27.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:31:27.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.891+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030a10 con 0x7f21cc042f20 2021-03-09T10:31:27.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.895+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12366 cs=0 l=1 rev1=1 rx=0x7f21cc004eb0 tx=0x7f21dc087270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:27.901 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:27.899+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42365 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:27.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3330 con 0x7f21cc042f20 2021-03-09T10:31:27.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3330 con 0x7f21cc042f20 2021-03-09T10:31:27.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.903+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc042f20 2021-03-09T10:31:27.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.903+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:27.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:27.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:27.903+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:32.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:32.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:32.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12366 cs=0 l=1 rev1=1 rx=0x7f21cc004eb0 tx=0x7f21dc087270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:32.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 secure :-1 s=READY pgs=12366 cs=0 l=1 rev1=1 rx=0x7f21cc004eb0 tx=0x7f21dc087270).stop 2021-03-09T10:31:32.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:32.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=12366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:32.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:32.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cd10 con 0x7f21cc005e30 2021-03-09T10:31:32.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016f10 con 0x7f21c4003cb0 2021-03-09T10:31:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.899+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021d30 con 0x7f21c4005da0 2021-03-09T10:31:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:32.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:32.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:32.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0889a0 con 0x7f21c4005da0 2021-03-09T10:31:32.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093fa0 con 0x7f21c4005da0 2021-03-09T10:31:32.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.903+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12376 cs=0 l=1 rev1=1 rx=0x7f21cc084a60 tx=0x7f21e4055590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:32.911 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:32.907+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42404 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:32.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:31:32.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:31:32.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:31:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.911+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:32.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.915+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:32.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:32.915+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:37.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:37.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:37.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12376 cs=0 l=1 rev1=1 rx=0x7f21cc084a60 tx=0x7f21e4055590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:37.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12376 cs=0 l=1 rev1=1 rx=0x7f21cc084a60 tx=0x7f21e4055590).stop 2021-03-09T10:31:37.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:37.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=CLOSED pgs=12376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:37.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21cc005e30 2021-03-09T10:31:37.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097730 con 0x7f21c4003cb0 2021-03-09T10:31:37.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cd10 con 0x7f21cc042f20 2021-03-09T10:31:37.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:37.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:37.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:37.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:37.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:37.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:37.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:37.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:37.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:37.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc084af0 con 0x7f21cc042f20 2021-03-09T10:31:37.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc042f20 2021-03-09T10:31:37.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.911+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12387 cs=0 l=1 rev1=1 rx=0x7f21e00a3f60 tx=0x7f21e00067f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:37.920 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:37.915+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42440 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:37.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.919+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0014e90 con 0x7f21cc042f20 2021-03-09T10:31:37.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0014e90 con 0x7f21cc042f20 2021-03-09T10:31:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:31:37.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.927+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.927+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:37.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:37.927+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:42.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:42.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:42.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12387 cs=0 l=1 rev1=1 rx=0x7f21e00a3f60 tx=0x7f21e00067f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:42.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12387 cs=0 l=1 rev1=1 rx=0x7f21e00a3f60 tx=0x7f21e00067f0).stop 2021-03-09T10:31:42.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=CLOSED pgs=12387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:42.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:42.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21cc005e30 2021-03-09T10:31:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cd10 con 0x7f21c4003cb0 2021-03-09T10:31:42.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec50 con 0x7f21c4005da0 2021-03-09T10:31:42.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:42.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:42.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:42.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:42.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.919+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:42.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:42.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:42.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:42.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:42.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021d30 con 0x7f21c4005da0 2021-03-09T10:31:42.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0889a0 con 0x7f21c4005da0 2021-03-09T10:31:42.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.923+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12400 cs=0 l=1 rev1=1 rx=0x7f21cc053690 tx=0x7f21e4016160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:42.935 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:42.927+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42479 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4005da0 2021-03-09T10:31:42.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.931+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21c4005da0 2021-03-09T10:31:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4005da0 2021-03-09T10:31:42.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.935+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:42.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.935+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:42.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:42.935+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:47.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:47.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:47.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12400 cs=0 l=1 rev1=1 rx=0x7f21cc053690 tx=0x7f21e4016160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:47.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12400 cs=0 l=1 rev1=1 rx=0x7f21cc053690 tx=0x7f21e4016160).stop 2021-03-09T10:31:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=CLOSED pgs=12400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:47.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21cc005e30 2021-03-09T10:31:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf10 con 0x7f21c4003cb0 2021-03-09T10:31:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cd10 con 0x7f21cc042f20 2021-03-09T10:31:47.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:47.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:47.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:47.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:47.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.927+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:47.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:47.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021d30 con 0x7f21cc042f20 2021-03-09T10:31:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0889a0 con 0x7f21cc042f20 2021-03-09T10:31:47.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.931+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12408 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e40173c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:47.936 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:47.935+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42515 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:47.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21cc042f20 2021-03-09T10:31:47.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40312c0 con 0x7f21cc042f20 2021-03-09T10:31:47.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.939+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4031160 con 0x7f21cc042f20 2021-03-09T10:31:47.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.939+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:47.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.939+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:47.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:47.939+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:52.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12408 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e40173c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12408 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e40173c0).stop 2021-03-09T10:31:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=CLOSED pgs=12408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:52.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:52.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:52.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:52.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21cc005e30 2021-03-09T10:31:52.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041900 con 0x7f21c4003cb0 2021-03-09T10:31:52.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096a10 con 0x7f21c4005da0 2021-03-09T10:31:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:52.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:52.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.935+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:52.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:52.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc089830 con 0x7f21c4005da0 2021-03-09T10:31:52.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc084af0 con 0x7f21c4005da0 2021-03-09T10:31:52.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.939+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12419 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e002af90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:52.945 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:52.943+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:52.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.943+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4005da0 2021-03-09T10:31:52.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.943+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4005da0 2021-03-09T10:31:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.947+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21c4005da0 2021-03-09T10:31:52.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.947+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:52.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.947+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:52.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:52.947+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:31:56.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:57.000 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:31:57.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:57.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12419 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e002af90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 secure :-1 s=READY pgs=12419 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e002af90).stop 2021-03-09T10:31:57.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:57.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021840 unknown :-1 s=CLOSED pgs=12419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:57.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:57.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:57.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:57.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041900 con 0x7f21cc005e30 2021-03-09T10:31:57.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21c4003cb0 2021-03-09T10:31:57.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf10 con 0x7f21cc042f20 2021-03-09T10:31:57.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:31:57.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:57.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:31:57.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:57.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:57.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026160 con 0x7f21cc042f20 2021-03-09T10:31:57.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc089830 con 0x7f21cc042f20 2021-03-09T10:31:57.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.946+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12431 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e009d550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:57.954 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:31:57.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42590 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:31:57.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21cc042f20 2021-03-09T10:31:57.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000bc10 con 0x7f21cc042f20 2021-03-09T10:31:57.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.958+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc042f20 2021-03-09T10:31:57.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.958+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:31:57.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.962+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:31:57.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:31:57.962+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:02.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:32:03.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:03.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12431 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e009d550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:03.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12431 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e009d550).stop 2021-03-09T10:32:03.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:03.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=CLOSED pgs=12431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:03.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:03.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:03.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:03.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21cc005e30 2021-03-09T10:32:03.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041900 con 0x7f21c4003cb0 2021-03-09T10:32:03.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e80 con 0x7f21c4005da0 2021-03-09T10:32:03.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:03.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:03.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:03.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:03.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:03.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:03.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:03.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:03.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:03.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04ec30 con 0x7f21c4005da0 2021-03-09T10:32:03.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.954+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:32:03.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.958+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12441 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc081120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:03.301 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:02.958+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42628 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:03.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:32:03.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:32:03.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.966+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0328b0 con 0x7f21c4005da0 2021-03-09T10:32:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.966+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:03.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:02.966+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:07.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:07.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:07.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12441 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc081120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:07.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12441 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc081120).stop 2021-03-09T10:32:07.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:07.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=CLOSED pgs=12441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:07.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:07.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:07.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:07.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041900 con 0x7f21cc005e30 2021-03-09T10:32:07.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21c4003cb0 2021-03-09T10:32:07.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf10 con 0x7f21cc042f20 2021-03-09T10:32:07.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:07.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.962+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:07.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc029e50 con 0x7f21cc042f20 2021-03-09T10:32:07.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04ec30 con 0x7f21cc042f20 2021-03-09T10:32:07.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.966+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12451 cs=0 l=1 rev1=1 rx=0x7f21dc00aef0 tx=0x7f21dc071f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:07.971 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:07.970+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42665 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:07.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00af60 con 0x7f21cc042f20 2021-03-09T10:32:07.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc00af60 con 0x7f21cc042f20 2021-03-09T10:32:07.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06a000 con 0x7f21cc042f20 2021-03-09T10:32:07.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.970+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:07.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.974+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:07.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:07.974+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:12.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:12.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:12.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12451 cs=0 l=1 rev1=1 rx=0x7f21dc00aef0 tx=0x7f21dc071f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:12.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12451 cs=0 l=1 rev1=1 rx=0x7f21dc00aef0 tx=0x7f21dc071f10).stop 2021-03-09T10:32:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=CLOSED pgs=12451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:12.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035d90 con 0x7f21cc005e30 2021-03-09T10:32:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0266e0 con 0x7f21c4003cb0 2021-03-09T10:32:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041900 con 0x7f21c4005da0 2021-03-09T10:32:12.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:12.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:12.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:12.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:12.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:12.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:12.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04ec30 con 0x7f21c4005da0 2021-03-09T10:32:12.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4005da0 2021-03-09T10:32:12.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.970+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12462 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e007e4a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:12.979 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:12.978+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42703 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:12.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21c4005da0 2021-03-09T10:32:12.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e009d610 con 0x7f21c4005da0 2021-03-09T10:32:12.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.982+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003dde0 con 0x7f21c4005da0 2021-03-09T10:32:12.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.982+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:12.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.982+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:12.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:12.982+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:17.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:17.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:17.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12462 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e007e4a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:17.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12462 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e007e4a0).stop 2021-03-09T10:32:17.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:17.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=CLOSED pgs=12462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:17.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:17.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:17.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:17.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:17.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:17.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0266e0 con 0x7f21cc005e30 2021-03-09T10:32:17.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4003cb0 2021-03-09T10:32:17.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0215a0 con 0x7f21cc042f20 2021-03-09T10:32:17.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:17.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:17.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:17.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:17.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:17.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:17.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.978+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:17.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.982+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a850 con 0x7f21cc042f20 2021-03-09T10:32:17.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.982+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc029e50 con 0x7f21cc042f20 2021-03-09T10:32:17.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.982+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12472 cs=0 l=1 rev1=1 rx=0x7f21cc0415f0 tx=0x7f21dc07cc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:17.988 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:17.986+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42740 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:17.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21cc042f20 2021-03-09T10:32:17.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089950 con 0x7f21cc042f20 2021-03-09T10:32:17.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc089d40 con 0x7f21cc042f20 2021-03-09T10:32:17.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.994+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:17.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.994+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:17.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:17.994+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:22.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12472 cs=0 l=1 rev1=1 rx=0x7f21cc0415f0 tx=0x7f21dc07cc20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:22.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 secure :-1 s=READY pgs=12472 cs=0 l=1 rev1=1 rx=0x7f21cc0415f0 tx=0x7f21dc07cc20).stop 2021-03-09T10:32:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc088b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc088b30 unknown :-1 s=CLOSED pgs=12472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:22.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:22.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e50 con 0x7f21cc005e30 2021-03-09T10:32:22.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:22.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021d00 con 0x7f21c4003cb0 2021-03-09T10:32:22.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:32:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:22.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.986+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:22.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0215d0 con 0x7f21c4005da0 2021-03-09T10:32:22.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0889a0 con 0x7f21c4005da0 2021-03-09T10:32:22.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.990+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12485 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e40b5000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:22.996 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:22.994+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42778 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:22.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c4005da0 2021-03-09T10:32:22.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c4005da0 2021-03-09T10:32:23.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.998+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5dd0 con 0x7f21c4005da0 2021-03-09T10:32:23.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:22.998+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:23.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:23.002+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:23.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:23.002+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:27.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:27.057 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:27.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:27.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:27.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12485 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e40b5000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:27.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12485 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e40b5000).stop 2021-03-09T10:32:27.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:27.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=CLOSED pgs=12485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:27.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:27.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:28.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:28.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc005e30 2021-03-09T10:32:28.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e30 con 0x7f21c4003cb0 2021-03-09T10:32:28.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0889a0 con 0x7f21cc042f20 2021-03-09T10:32:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:28.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:28.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.994+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:28.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:28.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:28.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:28.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:28.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc088ae0 con 0x7f21cc042f20 2021-03-09T10:32:28.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04ec30 con 0x7f21cc042f20 2021-03-09T10:32:28.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:27.998+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0466f0 secure :-1 s=READY pgs=12496 cs=0 l=1 rev1=1 rx=0x7f21e00214c0 tx=0x7f21e00a32e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:28.006 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:28.002+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42815 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:28.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:28.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:32:28.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:28.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc042f20 2021-03-09T10:32:28.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:28.010+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003dde0 con 0x7f21cc042f20 2021-03-09T10:32:28.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:28.010+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:28.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:28.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:28.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:28.010+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:33.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0466f0 secure :-1 s=READY pgs=12496 cs=0 l=1 rev1=1 rx=0x7f21e00214c0 tx=0x7f21e00a32e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:33.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0466f0 secure :-1 s=READY pgs=12496 cs=0 l=1 rev1=1 rx=0x7f21e00214c0 tx=0x7f21e00a32e0).stop 2021-03-09T10:32:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0466f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0466f0 unknown :-1 s=CLOSED pgs=12496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:33.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0889a0 con 0x7f21cc005e30 2021-03-09T10:32:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c4003cb0 2021-03-09T10:32:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:32:33.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:33.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:33.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:33.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.006+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:33.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:33.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04ec30 con 0x7f21c4005da0 2021-03-09T10:32:33.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0215d0 con 0x7f21c4005da0 2021-03-09T10:32:33.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12507 cs=0 l=1 rev1=1 rx=0x7f21e407e580 tx=0x7f21e4053fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:33.014 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:33.010+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:33.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027eb0 con 0x7f21c4005da0 2021-03-09T10:32:33.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.018+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027eb0 con 0x7f21c4005da0 2021-03-09T10:32:33.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5dd0 con 0x7f21c4005da0 2021-03-09T10:32:33.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.022+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:33.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.022+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:33.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:33.022+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:38.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:38.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:38.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12507 cs=0 l=1 rev1=1 rx=0x7f21e407e580 tx=0x7f21e4053fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:38.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12507 cs=0 l=1 rev1=1 rx=0x7f21e407e580 tx=0x7f21e4053fb0).stop 2021-03-09T10:32:38.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:38.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=CLOSED pgs=12507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:38.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:38.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:38.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:38.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e3b0 con 0x7f21cc005e30 2021-03-09T10:32:38.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08de90 con 0x7f21c4003cb0 2021-03-09T10:32:38.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21cc042f20 2021-03-09T10:32:38.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:38.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:38.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:38.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:38.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.014+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:38.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:38.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:38.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:38.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:38.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0468a0 con 0x7f21cc042f20 2021-03-09T10:32:38.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04ec30 con 0x7f21cc042f20 2021-03-09T10:32:38.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.018+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0463b0 secure :-1 s=READY pgs=12516 cs=0 l=1 rev1=1 rx=0x7f21cc089800 tx=0x7f21e400de40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:38.024 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:38.022+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42890 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:38.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031360 con 0x7f21cc042f20 2021-03-09T10:32:38.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a3070 con 0x7f21cc042f20 2021-03-09T10:32:38.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.026+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a31e0 con 0x7f21cc042f20 2021-03-09T10:32:38.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.026+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:38.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:38.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:38.026+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:43.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:43.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:43.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0463b0 secure :-1 s=READY pgs=12516 cs=0 l=1 rev1=1 rx=0x7f21cc089800 tx=0x7f21e400de40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:43.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0463b0 secure :-1 s=READY pgs=12516 cs=0 l=1 rev1=1 rx=0x7f21cc089800 tx=0x7f21e400de40).stop 2021-03-09T10:32:43.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0463b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:43.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0463b0 unknown :-1 s=CLOSED pgs=12516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:43.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:43.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:43.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:43.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e030 con 0x7f21cc005e30 2021-03-09T10:32:43.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c4003cb0 2021-03-09T10:32:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e390 con 0x7f21c4005da0 2021-03-09T10:32:43.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:43.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:43.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:43.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:43.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:43.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:43.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:43.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:43.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:43.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04ec30 con 0x7f21c4005da0 2021-03-09T10:32:43.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.022+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc088ae0 con 0x7f21c4005da0 2021-03-09T10:32:43.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.026+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12526 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007c600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:43.043 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:43.030+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42928 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0004e50 con 0x7f21c4005da0 2021-03-09T10:32:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.030+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004e50 con 0x7f21c4005da0 2021-03-09T10:32:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.034+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003dde0 con 0x7f21c4005da0 2021-03-09T10:32:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.034+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:43.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.034+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:43.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:43.034+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:48.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:48.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:48.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12526 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007c600).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:48.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12526 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007c600).stop 2021-03-09T10:32:48.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:48.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=CLOSED pgs=12526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:48.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:48.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:48.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:48.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21cc005e30 2021-03-09T10:32:48.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08e030 con 0x7f21c4003cb0 2021-03-09T10:32:48.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e370 con 0x7f21cc042f20 2021-03-09T10:32:48.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:48.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:48.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:48.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:48.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:48.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.029+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc042f20 2021-03-09T10:32:48.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.033+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0468a0 con 0x7f21cc042f20 2021-03-09T10:32:48.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.033+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046300 secure :-1 s=READY pgs=12537 cs=0 l=1 rev1=1 rx=0x7f21cc08acc0 tx=0x7f21e40a6ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:48.040 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:48.037+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42965 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:48.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21cc042f20 2021-03-09T10:32:48.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407e330 con 0x7f21cc042f20 2021-03-09T10:32:48.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009270 con 0x7f21cc042f20 2021-03-09T10:32:48.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.045+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:48.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.045+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:48.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:48.045+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:53.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:53.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:53.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046300 secure :-1 s=READY pgs=12537 cs=0 l=1 rev1=1 rx=0x7f21cc08acc0 tx=0x7f21e40a6ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:53.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046300 secure :-1 s=READY pgs=12537 cs=0 l=1 rev1=1 rx=0x7f21cc08acc0 tx=0x7f21e40a6ad0).stop 2021-03-09T10:32:53.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc046300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:53.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc046300 unknown :-1 s=CLOSED pgs=12537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:53.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:53.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:53.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:53.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21cc005e30 2021-03-09T10:32:53.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f670 con 0x7f21c4003cb0 2021-03-09T10:32:53.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094570 con 0x7f21c4005da0 2021-03-09T10:32:53.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:53.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:53.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:53.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.037+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:53.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:53.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:53.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021d00 con 0x7f21c4005da0 2021-03-09T10:32:53.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.041+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:32:53.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.045+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12553 cs=0 l=1 rev1=1 rx=0x7f21cc017550 tx=0x7f21e4016ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:53.049 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:53.045+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43003 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:53.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d1c0 con 0x7f21c4005da0 2021-03-09T10:32:53.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d1c0 con 0x7f21c4005da0 2021-03-09T10:32:53.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21c4005da0 2021-03-09T10:32:53.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.057+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:53.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.057+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:53.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:53.057+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:32:57.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:57.103 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:32:58.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:58.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:58.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12553 cs=0 l=1 rev1=1 rx=0x7f21cc017550 tx=0x7f21e4016ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:58.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 secure :-1 s=READY pgs=12553 cs=0 l=1 rev1=1 rx=0x7f21cc017550 tx=0x7f21e4016ab0).stop 2021-03-09T10:32:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc021810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc021810 unknown :-1 s=CLOSED pgs=12553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:58.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.045+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21cc005e30 2021-03-09T10:32:58.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094570 con 0x7f21c4003cb0 2021-03-09T10:32:58.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021d00 con 0x7f21cc042f20 2021-03-09T10:32:58.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:58.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:58.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:58.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:58.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:58.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 msgr2=0x7f21cc0462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:32:58.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc042f20 0x7f21cc0462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:58.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 msgr2=0x7f21cc0165b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:58.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c4003cb0 0x7f21cc0165b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:58.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 msgr2=0x7f21cc039b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:58.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21cc005e30 0x7f21cc039b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:58.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21c4004130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:58.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21c400a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:58.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21c4004aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:58.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c400a6b0 2021-03-09T10:32:58.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094570 con 0x7f21c40037a0 2021-03-09T10:32:58.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21e97fa700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021d00 con 0x7f21c4004620 2021-03-09T10:32:58.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21c4004aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:32:58.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21c4004aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:58.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21c400a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:58.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21c4004130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:32:58.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.049+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21c4004130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:58.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21c4004130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:58.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21c4004130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:58.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21c4004aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:58.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21c4004aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:58.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0467a0 con 0x7f21c4004620 2021-03-09T10:32:58.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:32:58.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21c400a1c0 secure :-1 s=READY pgs=12565 cs=0 l=1 rev1=1 rx=0x7f21e400dda0 tx=0x7f21e4055440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:58.063 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:32:58.053+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43040 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:32:58.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f2e0 con 0x7f21c4004620 2021-03-09T10:32:58.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:32:58.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.061+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407f2e0 con 0x7f21c4004620 2021-03-09T10:32:58.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.061+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:32:58.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.061+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:32:58.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:32:58.061+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:03.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21c400a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:03.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21c400a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:03.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21c400a1c0 secure :-1 s=READY pgs=12565 cs=0 l=1 rev1=1 rx=0x7f21e400dda0 tx=0x7f21e4055440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:03.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21c400a1c0 secure :-1 s=READY pgs=12565 cs=0 l=1 rev1=1 rx=0x7f21e400dda0 tx=0x7f21e4055440).stop 2021-03-09T10:33:03.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21c400a1c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:03.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21c400a1c0 unknown :-1 s=CLOSED pgs=12565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:03.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:03.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:03.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:03.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c400a6b0 2021-03-09T10:33:03.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094570 con 0x7f21c40037a0 2021-03-09T10:33:03.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f7c0 con 0x7f21c4005da0 2021-03-09T10:33:03.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:03.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:03.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:03.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc084af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:03.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:03.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:03.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:03.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:03.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc084af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:03.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:33:03.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.057+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0467a0 con 0x7f21c4005da0 2021-03-09T10:33:03.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.061+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0399b0 secure :-1 s=READY pgs=12570 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4041300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:03.066 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:03.061+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43078 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:03.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e403cc80 con 0x7f21c4005da0 2021-03-09T10:33:03.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e403cc80 con 0x7f21c4005da0 2021-03-09T10:33:03.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402b710 con 0x7f21c4005da0 2021-03-09T10:33:03.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.073+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:03.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.073+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:03.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:03.073+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0399b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0399b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:08.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0399b0 secure :-1 s=READY pgs=12570 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4041300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0399b0 secure :-1 s=READY pgs=12570 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e4041300).stop 2021-03-09T10:33:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0399b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0399b0 unknown :-1 s=CLOSED pgs=12570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:08.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc042d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d7c0 con 0x7f21c400a6b0 2021-03-09T10:33:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c40037a0 2021-03-09T10:33:08.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094570 con 0x7f21c4004620 2021-03-09T10:33:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:08.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:08.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc042d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:08.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:08.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:08.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:08.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:08.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0431f0 con 0x7f21c4004620 2021-03-09T10:33:08.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.065+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0467a0 con 0x7f21c4004620 2021-03-09T10:33:08.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.069+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc042d00 secure :-1 s=READY pgs=12580 cs=0 l=1 rev1=1 rx=0x7f21e40b02a0 tx=0x7f21e403cc80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:08.075 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:08.073+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43115 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:08.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:33:08.077 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c4004620 2021-03-09T10:33:08.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.077+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:33:08.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.077+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:08.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.077+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:08.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:08.077+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:13.076 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc042d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc042d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc042d00 secure :-1 s=READY pgs=12580 cs=0 l=1 rev1=1 rx=0x7f21e40b02a0 tx=0x7f21e403cc80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:13.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc042d00 secure :-1 s=READY pgs=12580 cs=0 l=1 rev1=1 rx=0x7f21e40b02a0 tx=0x7f21e403cc80).stop 2021-03-09T10:33:13.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc042d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:13.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc042d00 unknown :-1 s=CLOSED pgs=12580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:13.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:13.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0397c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:13.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:13.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05a620 con 0x7f21c400a6b0 2021-03-09T10:33:13.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094570 con 0x7f21c40037a0 2021-03-09T10:33:13.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:33:13.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:13.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:13.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d0b0 con 0x7f21c4005da0 2021-03-09T10:33:13.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0431f0 con 0x7f21c4005da0 2021-03-09T10:33:13.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.073+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0397c0 secure :-1 s=READY pgs=12590 cs=0 l=1 rev1=1 rx=0x7f21e40b04d0 tx=0x7f21e40a62a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:13.285 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:13.081+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43153 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:13.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4038ea0 con 0x7f21c4005da0 2021-03-09T10:33:13.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4005da0 2021-03-09T10:33:13.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.085+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5dd0 con 0x7f21c4005da0 2021-03-09T10:33:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.085+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.085+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:13.085+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:18.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:18.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:18.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0397c0 secure :-1 s=READY pgs=12590 cs=0 l=1 rev1=1 rx=0x7f21e40b04d0 tx=0x7f21e40a62a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:18.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0397c0 secure :-1 s=READY pgs=12590 cs=0 l=1 rev1=1 rx=0x7f21e40b04d0 tx=0x7f21e40a62a0).stop 2021-03-09T10:33:18.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0397c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:18.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0397c0 unknown :-1 s=CLOSED pgs=12590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:18.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:18.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:18.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:18.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c400a6b0 2021-03-09T10:33:18.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05a620 con 0x7f21c40037a0 2021-03-09T10:33:18.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042b10 con 0x7f21c4004620 2021-03-09T10:33:18.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:18.088 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:18.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:18.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:18.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:18.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:18.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042c50 con 0x7f21c4004620 2021-03-09T10:33:18.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.081+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04ec30 con 0x7f21c4004620 2021-03-09T10:33:18.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.085+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039cb0 secure :-1 s=READY pgs=12600 cs=0 l=1 rev1=1 rx=0x7f21e0007810 tx=0x7f21e007cb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:18.093 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:18.089+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43190 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000b3e0 con 0x7f21c4004620 2021-03-09T10:33:18.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0007470 con 0x7f21c4004620 2021-03-09T10:33:18.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.093+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003dde0 con 0x7f21c4004620 2021-03-09T10:33:18.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.093+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:18.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:18.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:18.093+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:23.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:23.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039cb0 secure :-1 s=READY pgs=12600 cs=0 l=1 rev1=1 rx=0x7f21e0007810 tx=0x7f21e007cb20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039cb0 secure :-1 s=READY pgs=12600 cs=0 l=1 rev1=1 rx=0x7f21e0007810 tx=0x7f21e007cb20).stop 2021-03-09T10:33:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:23.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039cb0 unknown :-1 s=CLOSED pgs=12600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:23.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c400a6b0 2021-03-09T10:33:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b760 con 0x7f21c40037a0 2021-03-09T10:33:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05a620 con 0x7f21c4005da0 2021-03-09T10:33:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:23.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:23.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:23.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.089+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:23.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:23.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:23.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042ac0 con 0x7f21c4005da0 2021-03-09T10:33:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042c50 con 0x7f21c4005da0 2021-03-09T10:33:23.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.093+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039750 secure :-1 s=READY pgs=12612 cs=0 l=1 rev1=1 rx=0x7f21e0059740 tx=0x7f21e00a6190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:23.100 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:23.097+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43228 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21c4005da0 2021-03-09T10:33:23.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21c4005da0 2021-03-09T10:33:23.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.101+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0059000 con 0x7f21c4005da0 2021-03-09T10:33:23.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.101+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:23.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:23.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:23.101+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:27.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:27.151 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:28.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:28.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:28.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039750 secure :-1 s=READY pgs=12612 cs=0 l=1 rev1=1 rx=0x7f21e0059740 tx=0x7f21e00a6190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:28.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039750 secure :-1 s=READY pgs=12612 cs=0 l=1 rev1=1 rx=0x7f21e0059740 tx=0x7f21e00a6190).stop 2021-03-09T10:33:28.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:28.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039750 unknown :-1 s=CLOSED pgs=12612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05a620 con 0x7f21c400a6b0 2021-03-09T10:33:28.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094120 con 0x7f21c40037a0 2021-03-09T10:33:28.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094ef0 con 0x7f21c4004620 2021-03-09T10:33:28.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:28.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:28.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:28.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.097+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:28.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:28.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:28.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:28.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:28.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:28.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095030 con 0x7f21c4004620 2021-03-09T10:33:28.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d0b0 con 0x7f21c4004620 2021-03-09T10:33:28.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.101+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c40 secure :-1 s=READY pgs=12624 cs=0 l=1 rev1=1 rx=0x7f21e402d910 tx=0x7f21e409bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:28.109 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:28.105+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43265 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:28.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21c4004620 2021-03-09T10:33:28.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.113+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21c4004620 2021-03-09T10:33:28.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.113+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016c80 con 0x7f21c4004620 2021-03-09T10:33:28.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.113+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:28.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.117+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:28.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:28.117+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:33.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:33.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:33.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c40 secure :-1 s=READY pgs=12624 cs=0 l=1 rev1=1 rx=0x7f21e402d910 tx=0x7f21e409bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:33.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c40 secure :-1 s=READY pgs=12624 cs=0 l=1 rev1=1 rx=0x7f21e402d910 tx=0x7f21e409bbb0).stop 2021-03-09T10:33:33.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:33.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c40 unknown :-1 s=CLOSED pgs=12624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:33.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:33.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:33.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:33.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094120 con 0x7f21c400a6b0 2021-03-09T10:33:33.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d2d0 con 0x7f21c40037a0 2021-03-09T10:33:33.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d410 con 0x7f21c4005da0 2021-03-09T10:33:33.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:33.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:33.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:33.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:33.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:33.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:33.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:33.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:33.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:33.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02d550 con 0x7f21c4005da0 2021-03-09T10:33:33.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095030 con 0x7f21c4005da0 2021-03-09T10:33:33.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.109+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039710 secure :-1 s=READY pgs=12632 cs=0 l=1 rev1=1 rx=0x7f21e404f520 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:33.117 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:33.113+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43303 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:33.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.113+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4005da0 2021-03-09T10:33:33.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.121+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21c4005da0 2021-03-09T10:33:33.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.121+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4005da0 2021-03-09T10:33:33.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.121+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:33.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.121+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:33.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:33.121+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:38.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039710 secure :-1 s=READY pgs=12632 cs=0 l=1 rev1=1 rx=0x7f21e404f520 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039710 secure :-1 s=READY pgs=12632 cs=0 l=1 rev1=1 rx=0x7f21e404f520 tx=0x7f21e4053ed0).stop 2021-03-09T10:33:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039710 unknown :-1 s=CLOSED pgs=12632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:38.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041d70 con 0x7f21c400a6b0 2021-03-09T10:33:38.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094120 con 0x7f21c40037a0 2021-03-09T10:33:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042c50 con 0x7f21c4004620 2021-03-09T10:33:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:38.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:38.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:38.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:38.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:38.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:38.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:38.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042d90 con 0x7f21c4004620 2021-03-09T10:33:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042ac0 con 0x7f21c4004620 2021-03-09T10:33:38.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c00 secure :-1 s=READY pgs=12643 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e007cb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:38.126 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:38.120+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43340 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:38.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21c4004620 2021-03-09T10:33:38.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a3000 con 0x7f21c4004620 2021-03-09T10:33:38.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00338d0 con 0x7f21c4004620 2021-03-09T10:33:38.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.124+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:38.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.128+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:38.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:38.128+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:43.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c00 secure :-1 s=READY pgs=12643 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e007cb20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:43.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c00 secure :-1 s=READY pgs=12643 cs=0 l=1 rev1=1 rx=0x7f21e007d420 tx=0x7f21e007cb20).stop 2021-03-09T10:33:43.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc039c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:43.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc039c00 unknown :-1 s=CLOSED pgs=12643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:43.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:43.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:43.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:43.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042c30 con 0x7f21c400a6b0 2021-03-09T10:33:43.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094120 con 0x7f21c40037a0 2021-03-09T10:33:43.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0398c0 con 0x7f21c4005da0 2021-03-09T10:33:43.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:43.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.120+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:43.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:43.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:43.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:43.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:43.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:43.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:43.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc021460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:43.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039a30 con 0x7f21c4005da0 2021-03-09T10:33:43.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02d550 con 0x7f21c4005da0 2021-03-09T10:33:43.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.124+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0393d0 secure :-1 s=READY pgs=12655 cs=0 l=1 rev1=1 rx=0x7f21cc041cf0 tx=0x7f21e4039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:43.134 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:43.128+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43378 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:43.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4005da0 2021-03-09T10:33:43.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4005da0 2021-03-09T10:33:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a7280 con 0x7f21c4005da0 2021-03-09T10:33:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.140+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:43.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.140+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:43.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:43.140+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:48.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:48.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:48.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0393d0 secure :-1 s=READY pgs=12655 cs=0 l=1 rev1=1 rx=0x7f21cc041cf0 tx=0x7f21e4039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:48.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0393d0 secure :-1 s=READY pgs=12655 cs=0 l=1 rev1=1 rx=0x7f21cc041cf0 tx=0x7f21e4039000).stop 2021-03-09T10:33:48.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0393d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:48.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0393d0 unknown :-1 s=CLOSED pgs=12655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:48.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:48.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:48.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:48.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026160 con 0x7f21c400a6b0 2021-03-09T10:33:48.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047690 con 0x7f21c40037a0 2021-03-09T10:33:48.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042c30 con 0x7f21c4004620 2021-03-09T10:33:48.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:48.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:48.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:48.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:48.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:48.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:48.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:48.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:48.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:48.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094120 con 0x7f21c4004620 2021-03-09T10:33:48.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.132+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039a30 con 0x7f21c4004620 2021-03-09T10:33:48.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12665 cs=0 l=1 rev1=1 rx=0x7f21e40b4520 tx=0x7f21e401a320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:48.142 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:48.136+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43415 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:48.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b4390 con 0x7f21c4004620 2021-03-09T10:33:48.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4015000 con 0x7f21c4004620 2021-03-09T10:33:48.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e402c250 con 0x7f21c4004620 2021-03-09T10:33:48.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.140+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:48.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.144+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:48.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:48.144+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:53.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:53.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:53.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12665 cs=0 l=1 rev1=1 rx=0x7f21e40b4520 tx=0x7f21e401a320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:53.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12665 cs=0 l=1 rev1=1 rx=0x7f21e40b4520 tx=0x7f21e401a320).stop 2021-03-09T10:33:53.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:53.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:53.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:53.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:53.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:53.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026160 con 0x7f21c400a6b0 2021-03-09T10:33:53.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042c30 con 0x7f21c40037a0 2021-03-09T10:33:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022d50 con 0x7f21c4005da0 2021-03-09T10:33:53.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:53.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:53.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:53.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:53.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:53.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039d40 con 0x7f21c4005da0 2021-03-09T10:33:53.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a850 con 0x7f21c4005da0 2021-03-09T10:33:53.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.140+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12676 cs=0 l=1 rev1=1 rx=0x7f21cc007e80 tx=0x7f21dc0811f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:53.155 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:53.148+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43453 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:53.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:33:53.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:33:53.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4005da0 2021-03-09T10:33:53.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.156+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:53.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.156+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:53.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:53.156+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:33:57.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:57.222 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:33:58.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:58.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:58.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12676 cs=0 l=1 rev1=1 rx=0x7f21cc007e80 tx=0x7f21dc0811f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:58.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12676 cs=0 l=1 rev1=1 rx=0x7f21cc007e80 tx=0x7f21dc0811f0).stop 2021-03-09T10:33:58.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=CLOSED pgs=12676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:58.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057890 con 0x7f21c400a6b0 2021-03-09T10:33:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042f20 con 0x7f21c40037a0 2021-03-09T10:33:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043060 con 0x7f21c4004620 2021-03-09T10:33:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:33:58.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:58.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:33:58.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:58.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.148+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:58.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:58.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:58.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:58.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:58.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00e8d0 con 0x7f21c4004620 2021-03-09T10:33:58.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094120 con 0x7f21c4004620 2021-03-09T10:33:58.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.152+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12686 cs=0 l=1 rev1=1 rx=0x7f21e4007500 tx=0x7f21e404f230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:58.161 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:33:58.156+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43489 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:33:58.161 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4053c50 con 0x7f21c4004620 2021-03-09T10:33:58.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4015000 con 0x7f21c4004620 2021-03-09T10:33:58.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.160+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5dd0 con 0x7f21c4004620 2021-03-09T10:33:58.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.160+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:33:58.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.160+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:33:58.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:33:58.160+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:03.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:03.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:03.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12686 cs=0 l=1 rev1=1 rx=0x7f21e4007500 tx=0x7f21e404f230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:03.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12686 cs=0 l=1 rev1=1 rx=0x7f21e4007500 tx=0x7f21e404f230).stop 2021-03-09T10:34:03.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:03.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:03.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:03.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:03.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:03.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ec00 con 0x7f21c400a6b0 2021-03-09T10:34:03.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026ee0 con 0x7f21c40037a0 2021-03-09T10:34:03.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042f20 con 0x7f21c4005da0 2021-03-09T10:34:03.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:03.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:03.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:03.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:03.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:03.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:03.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:03.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:03.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:03.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043060 con 0x7f21c4005da0 2021-03-09T10:34:03.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00e8d0 con 0x7f21c4005da0 2021-03-09T10:34:03.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.160+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12696 cs=0 l=1 rev1=1 rx=0x7f21cc017e90 tx=0x7f21e4033540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:03.170 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:03.164+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43527 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:03.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4005da0 2021-03-09T10:34:03.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4053c50 con 0x7f21c4005da0 2021-03-09T10:34:03.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4005da0 2021-03-09T10:34:03.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.168+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:03.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.168+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:03.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:03.168+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:08.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:08.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:08.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12696 cs=0 l=1 rev1=1 rx=0x7f21cc017e90 tx=0x7f21e4033540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:08.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12696 cs=0 l=1 rev1=1 rx=0x7f21cc017e90 tx=0x7f21e4033540).stop 2021-03-09T10:34:08.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:08.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=CLOSED pgs=12696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:08.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:08.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042f20 con 0x7f21c400a6b0 2021-03-09T10:34:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089210 con 0x7f21c40037a0 2021-03-09T10:34:08.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093f80 con 0x7f21c4004620 2021-03-09T10:34:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:08.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:08.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:08.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0940c0 con 0x7f21c4004620 2021-03-09T10:34:08.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042d90 con 0x7f21c4004620 2021-03-09T10:34:08.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.168+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12707 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e002af50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:08.179 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:08.176+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43564 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:08.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4004620 2021-03-09T10:34:08.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.176+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003cda0 con 0x7f21c4004620 2021-03-09T10:34:08.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:34:08.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.180+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:08.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.180+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:08.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:08.180+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:13.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:13.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12707 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e002af50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12707 cs=0 l=1 rev1=1 rx=0x7f21e009e010 tx=0x7f21e002af50).stop 2021-03-09T10:34:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:13.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:13.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:13.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:13.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042d90 con 0x7f21c400a6b0 2021-03-09T10:34:13.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ed0 con 0x7f21c40037a0 2021-03-09T10:34:13.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b110 con 0x7f21c4005da0 2021-03-09T10:34:13.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:13.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:13.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:13.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:13.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:13.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:13.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:13.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:13.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:13.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04b280 con 0x7f21c4005da0 2021-03-09T10:34:13.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043060 con 0x7f21c4005da0 2021-03-09T10:34:13.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.180+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12718 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e409bda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:13.188 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:13.184+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43601 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:13.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4005da0 2021-03-09T10:34:13.189 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.184+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4005da0 2021-03-09T10:34:13.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5dd0 con 0x7f21c4005da0 2021-03-09T10:34:13.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.192+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:13.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:13.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:13.192+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:18.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12718 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e409bda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:18.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12718 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e409bda0).stop 2021-03-09T10:34:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=CLOSED pgs=12718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:18.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015d50 con 0x7f21c400a6b0 2021-03-09T10:34:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042d90 con 0x7f21c40037a0 2021-03-09T10:34:18.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ed0 con 0x7f21c4004620 2021-03-09T10:34:18.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:18.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:18.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:18.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:18.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:18.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:18.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:18.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:18.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:18.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043010 con 0x7f21c4004620 2021-03-09T10:34:18.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039d40 con 0x7f21c4004620 2021-03-09T10:34:18.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.188+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12728 cs=0 l=1 rev1=1 rx=0x7f21cc096250 tx=0x7f21dc0abeb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:18.198 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:18.192+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43639 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:18.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0695d0 con 0x7f21c4004620 2021-03-09T10:34:18.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.192+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0695d0 con 0x7f21c4004620 2021-03-09T10:34:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c4004620 2021-03-09T10:34:18.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.196+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:18.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.196+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:18.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:18.196+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:23.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:23.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:23.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12728 cs=0 l=1 rev1=1 rx=0x7f21cc096250 tx=0x7f21dc0abeb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:23.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12728 cs=0 l=1 rev1=1 rx=0x7f21cc096250 tx=0x7f21dc0abeb0).stop 2021-03-09T10:34:23.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:23.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=CLOSED pgs=12728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:23.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:23.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:23.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:23.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc015ef0 con 0x7f21c400a6b0 2021-03-09T10:34:23.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039d40 con 0x7f21c40037a0 2021-03-09T10:34:23.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039e80 con 0x7f21c4005da0 2021-03-09T10:34:23.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:23.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:23.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:23.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:23.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:23.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4005da0 2021-03-09T10:34:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.196+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0940c0 con 0x7f21c4005da0 2021-03-09T10:34:23.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.200+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12743 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e007d5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:23.206 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:23.200+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:23.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e009d000 con 0x7f21c4005da0 2021-03-09T10:34:23.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00595f0 con 0x7f21c4005da0 2021-03-09T10:34:23.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.208+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:34:23.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.208+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:23.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.208+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:23.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:23.208+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:27.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:27.403 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:28.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.200+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12743 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e007d5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:28.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12743 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e007d5f0).stop 2021-03-09T10:34:28.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:28.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=CLOSED pgs=12743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:28.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:28.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0940c0 con 0x7f21c400a6b0 2021-03-09T10:34:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00edc0 con 0x7f21c40037a0 2021-03-09T10:34:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a590 con 0x7f21c4004620 2021-03-09T10:34:28.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:28.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:28.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:28.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:28.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc029e30 con 0x7f21c4004620 2021-03-09T10:34:28.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:34:28.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.204+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12750 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:28.215 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:28.208+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43714 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:28.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.212+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0030 con 0x7f21c4004620 2021-03-09T10:34:28.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.212+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:34:28.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.212+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009d730 con 0x7f21c4004620 2021-03-09T10:34:28.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.212+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:28.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.216+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:28.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:28.216+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:33.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:33.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:33.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12750 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e000b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:33.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12750 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e000b040).stop 2021-03-09T10:34:33.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:33.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=CLOSED pgs=12750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:33.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:33.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:33.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:33.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a0c0 con 0x7f21c400a6b0 2021-03-09T10:34:33.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0940c0 con 0x7f21c40037a0 2021-03-09T10:34:33.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039d40 con 0x7f21c4005da0 2021-03-09T10:34:33.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:33.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:33.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:33.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:33.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:33.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:33.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:33.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:33.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:33.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a3f0 con 0x7f21c4005da0 2021-03-09T10:34:33.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4005da0 2021-03-09T10:34:33.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.211+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12760 cs=0 l=1 rev1=1 rx=0x7f21e0055900 tx=0x7f21e0039430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:33.224 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:33.219+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43751 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:33.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21c4005da0 2021-03-09T10:34:33.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004070 con 0x7f21c4005da0 2021-03-09T10:34:33.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091960 con 0x7f21c4005da0 2021-03-09T10:34:33.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.223+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:33.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.223+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:33.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:33.223+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:38.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:38.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:38.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12760 cs=0 l=1 rev1=1 rx=0x7f21e0055900 tx=0x7f21e0039430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:38.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 secure :-1 s=READY pgs=12760 cs=0 l=1 rev1=1 rx=0x7f21e0055900 tx=0x7f21e0039430).stop 2021-03-09T10:34:38.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc039850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:38.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc039850 unknown :-1 s=CLOSED pgs=12760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:38.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:38.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:38.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:38.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0940c0 con 0x7f21c400a6b0 2021-03-09T10:34:38.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00edc0 con 0x7f21c40037a0 2021-03-09T10:34:38.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041860 con 0x7f21c4004620 2021-03-09T10:34:38.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:38.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:38.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:38.229 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:38.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.223+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:38.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:38.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:38.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:38.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:38.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0419a0 con 0x7f21c4004620 2021-03-09T10:34:38.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a3f0 con 0x7f21c4004620 2021-03-09T10:34:38.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.227+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12774 cs=0 l=1 rev1=1 rx=0x7f21e00acfd0 tx=0x7f21e00a62b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:38.233 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:38.231+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43789 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:38.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21c4004620 2021-03-09T10:34:38.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21c4004620 2021-03-09T10:34:38.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.235+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009d730 con 0x7f21c4004620 2021-03-09T10:34:38.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.235+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:38.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.235+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:38.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:38.235+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:43.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:43.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:43.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12774 cs=0 l=1 rev1=1 rx=0x7f21e00acfd0 tx=0x7f21e00a62b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:43.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 secure :-1 s=READY pgs=12774 cs=0 l=1 rev1=1 rx=0x7f21e00acfd0 tx=0x7f21e00a62b0).stop 2021-03-09T10:34:43.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:43.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00e8d0 unknown :-1 s=CLOSED pgs=12774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:43.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc096f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:43.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:43.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0940c0 con 0x7f21c400a6b0 2021-03-09T10:34:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a590 con 0x7f21c40037a0 2021-03-09T10:34:43.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041810 con 0x7f21c4005da0 2021-03-09T10:34:43.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:43.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.231+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:43.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.235+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:43.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.235+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:43.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.235+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc096f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:43.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:43.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc039360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc039360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:43.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0973f0 con 0x7f21c4005da0 2021-03-09T10:34:43.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0419a0 con 0x7f21c4005da0 2021-03-09T10:34:43.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc096f00 secure :-1 s=READY pgs=12788 cs=0 l=1 rev1=1 rx=0x7f21e0037420 tx=0x7f21e001f6a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:43.247 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:43.239+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43826 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:43.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4005da0 2021-03-09T10:34:43.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4005da0 2021-03-09T10:34:43.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.247+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009a940 con 0x7f21c4005da0 2021-03-09T10:34:43.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.247+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:43.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.247+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:43.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:43.247+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:48.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc096f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:48.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc096f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc096f00 secure :-1 s=READY pgs=12788 cs=0 l=1 rev1=1 rx=0x7f21e0037420 tx=0x7f21e001f6a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc096f00 secure :-1 s=READY pgs=12788 cs=0 l=1 rev1=1 rx=0x7f21e0037420 tx=0x7f21e001f6a0).stop 2021-03-09T10:34:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc096f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:48.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc096f00 unknown :-1 s=CLOSED pgs=12788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:48.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0940c0 con 0x7f21c400a6b0 2021-03-09T10:34:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a590 con 0x7f21c40037a0 2021-03-09T10:34:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:34:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:48.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:48.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:48.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:48.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:48.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:48.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:48.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:48.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021a90 con 0x7f21c4004620 2021-03-09T10:34:48.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0973f0 con 0x7f21c4004620 2021-03-09T10:34:48.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.243+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12792 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e002b220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:48.251 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:48.247+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43864 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:48.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21c4004620 2021-03-09T10:34:48.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e004fce0 con 0x7f21c4004620 2021-03-09T10:34:48.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e009d730 con 0x7f21c4004620 2021-03-09T10:34:48.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.251+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:48.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.255+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:48.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:48.255+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:53.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:53.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:53.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12792 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e002b220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:53.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12792 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e002b220).stop 2021-03-09T10:34:53.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:53.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:53.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:53.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:53.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:53.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c400a6b0 2021-03-09T10:34:53.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0229a0 con 0x7f21c40037a0 2021-03-09T10:34:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ee0 con 0x7f21c4005da0 2021-03-09T10:34:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:53.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:53.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:53.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:53.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:53.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:53.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:53.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:53.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0940c0 con 0x7f21c4005da0 2021-03-09T10:34:53.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043010 con 0x7f21c4005da0 2021-03-09T10:34:53.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.255+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce90 secure :-1 s=READY pgs=12805 cs=0 l=1 rev1=1 rx=0x7f21cc007fe0 tx=0x7f21dc078490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:53.259 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:53.259+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43901 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:53.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0788a0 con 0x7f21c4005da0 2021-03-09T10:34:53.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc078670 con 0x7f21c4005da0 2021-03-09T10:34:53.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0787e0 con 0x7f21c4005da0 2021-03-09T10:34:53.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.267+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:53.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.267+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:53.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:53.267+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:34:57.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:57.444 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:34:58.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:58.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:58.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce90 secure :-1 s=READY pgs=12805 cs=0 l=1 rev1=1 rx=0x7f21cc007fe0 tx=0x7f21dc078490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:58.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce90 secure :-1 s=READY pgs=12805 cs=0 l=1 rev1=1 rx=0x7f21cc007fe0 tx=0x7f21dc078490).stop 2021-03-09T10:34:58.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04ce90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:58.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce90 unknown :-1 s=CLOSED pgs=12805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:58.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:58.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:58.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:58.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041530 con 0x7f21c400a6b0 2021-03-09T10:34:58.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039bb0 con 0x7f21c40037a0 2021-03-09T10:34:58.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0229a0 con 0x7f21c4004620 2021-03-09T10:34:58.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:34:58.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:58.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:58.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:34:58.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:58.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:58.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:58.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:58.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:58.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a590 con 0x7f21c4004620 2021-03-09T10:34:58.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04b280 con 0x7f21c4004620 2021-03-09T10:34:58.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.259+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12814 cs=0 l=1 rev1=1 rx=0x7f21e40a8050 tx=0x7f21e4005ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:58.267 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:34:58.267+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43939 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:34:58.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a5ea0 con 0x7f21c4004620 2021-03-09T10:34:58.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4004620 2021-03-09T10:34:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.271+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055cf0 con 0x7f21c4004620 2021-03-09T10:34:58.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.271+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:34:58.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.271+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:34:58.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:34:58.271+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:03.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:03.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:03.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12814 cs=0 l=1 rev1=1 rx=0x7f21e40a8050 tx=0x7f21e4005ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:03.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12814 cs=0 l=1 rev1=1 rx=0x7f21e40a8050 tx=0x7f21e4005ec0).stop 2021-03-09T10:35:03.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:03.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:03.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:03.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:03.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:03.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c400a6b0 2021-03-09T10:35:03.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017300 con 0x7f21c40037a0 2021-03-09T10:35:03.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0229a0 con 0x7f21c4005da0 2021-03-09T10:35:03.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:03.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:03.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:03.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:03.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.267+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:03.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:03.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:03.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04d300 con 0x7f21c4005da0 2021-03-09T10:35:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a590 con 0x7f21c4005da0 2021-03-09T10:35:03.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.271+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce10 secure :-1 s=READY pgs=12824 cs=0 l=1 rev1=1 rx=0x7f21cc04ec70 tx=0x7f21e4055cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:03.276 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:03.275+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43977 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:03.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21c4005da0 2021-03-09T10:35:03.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4005da0 2021-03-09T10:35:03.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21c4005da0 2021-03-09T10:35:03.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.283+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:03.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.283+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:03.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:03.283+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:08.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:08.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:08.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce10 secure :-1 s=READY pgs=12824 cs=0 l=1 rev1=1 rx=0x7f21cc04ec70 tx=0x7f21e4055cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:08.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce10 secure :-1 s=READY pgs=12824 cs=0 l=1 rev1=1 rx=0x7f21cc04ec70 tx=0x7f21e4055cf0).stop 2021-03-09T10:35:08.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04ce10 unknown :-1 s=CLOSED pgs=12824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:08.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c400a6b0 2021-03-09T10:35:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc007e30 con 0x7f21c40037a0 2021-03-09T10:35:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0409f0 con 0x7f21c4004620 2021-03-09T10:35:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:08.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:08.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:08.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:08.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.275+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:08.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:08.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:08.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:08.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:08.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017300 con 0x7f21c4004620 2021-03-09T10:35:08.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04d300 con 0x7f21c4004620 2021-03-09T10:35:08.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.279+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12835 cs=0 l=1 rev1=1 rx=0x7f21e4036f30 tx=0x7f21e4016180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:08.285 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:08.283+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44015 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:08.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:35:08.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4036d00 con 0x7f21c4004620 2021-03-09T10:35:08.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.287+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:35:08.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.287+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:08.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.291+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:08.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:08.291+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:13.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:13.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:13.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12835 cs=0 l=1 rev1=1 rx=0x7f21e4036f30 tx=0x7f21e4016180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:13.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12835 cs=0 l=1 rev1=1 rx=0x7f21e4036f30 tx=0x7f21e4016180).stop 2021-03-09T10:35:13.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:13.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:13.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:13.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:13.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:13.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0409f0 con 0x7f21c400a6b0 2021-03-09T10:35:13.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049d80 con 0x7f21c40037a0 2021-03-09T10:35:13.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049ec0 con 0x7f21c4005da0 2021-03-09T10:35:13.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:13.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:13.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:13.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:13.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.283+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:13.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:13.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:13.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:13.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:13.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a590 con 0x7f21c4005da0 2021-03-09T10:35:13.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017300 con 0x7f21c4005da0 2021-03-09T10:35:13.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.287+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 secure :-1 s=READY pgs=12849 cs=0 l=1 rev1=1 rx=0x7f21e40b4750 tx=0x7f21e4053c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:13.295 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:13.295+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44053 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:13.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404e070 con 0x7f21c4005da0 2021-03-09T10:35:13.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.299+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404e070 con 0x7f21c4005da0 2021-03-09T10:35:13.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f1d0 con 0x7f21c4005da0 2021-03-09T10:35:13.303 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.303+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:13.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:13.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:13.303+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:18.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:18.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:18.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 secure :-1 s=READY pgs=12849 cs=0 l=1 rev1=1 rx=0x7f21e40b4750 tx=0x7f21e4053c80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:18.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 secure :-1 s=READY pgs=12849 cs=0 l=1 rev1=1 rx=0x7f21e40b4750 tx=0x7f21e4053c80).stop 2021-03-09T10:35:18.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0416d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:18.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0416d0 unknown :-1 s=CLOSED pgs=12849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:18.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:18.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:18.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:18.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017300 con 0x7f21c400a6b0 2021-03-09T10:35:18.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049d80 con 0x7f21c40037a0 2021-03-09T10:35:18.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049ec0 con 0x7f21c4004620 2021-03-09T10:35:18.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:18.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:18.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:18.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:18.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:18.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:18.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:18.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:18.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:18.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a030 con 0x7f21c4004620 2021-03-09T10:35:18.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a590 con 0x7f21c4004620 2021-03-09T10:35:18.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.295+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12856 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e4017350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:18.304 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:18.303+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44091 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:18.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404e070 con 0x7f21c4004620 2021-03-09T10:35:18.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.303+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404e070 con 0x7f21c4004620 2021-03-09T10:35:18.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.307+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4062080 con 0x7f21c4004620 2021-03-09T10:35:18.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.307+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:18.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.307+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:18.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:18.307+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:23.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:35:23.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:23.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12856 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e4017350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:23.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12856 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e4017350).stop 2021-03-09T10:35:23.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:23.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:23.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:23.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:23.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:23.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c400a6b0 2021-03-09T10:35:23.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017300 con 0x7f21c40037a0 2021-03-09T10:35:23.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a590 con 0x7f21c4005da0 2021-03-09T10:35:23.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:23.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:23.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:23.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:23.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:23.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21c4005da0 2021-03-09T10:35:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021a90 con 0x7f21c4005da0 2021-03-09T10:35:23.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.306+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12867 cs=0 l=1 rev1=1 rx=0x7f21e00a46a0 tx=0x7f21e007fd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:23.312 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:23.310+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44129 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:23.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6210 con 0x7f21c4005da0 2021-03-09T10:35:23.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003e380 con 0x7f21c4005da0 2021-03-09T10:35:23.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003cca0 con 0x7f21c4005da0 2021-03-09T10:35:23.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.314+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:23.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.314+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:23.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:23.314+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:27.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:27.490 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:28.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:28.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12867 cs=0 l=1 rev1=1 rx=0x7f21e00a46a0 tx=0x7f21e007fd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12867 cs=0 l=1 rev1=1 rx=0x7f21e00a46a0 tx=0x7f21e007fd20).stop 2021-03-09T10:35:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:28.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=12867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:28.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:28.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:28.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:28.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095290 con 0x7f21c400a6b0 2021-03-09T10:35:28.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022e60 con 0x7f21c40037a0 2021-03-09T10:35:28.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a1c0 con 0x7f21c4004620 2021-03-09T10:35:28.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:28.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:28.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:28.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.310+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:28.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:28.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:28.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017300 con 0x7f21c4004620 2021-03-09T10:35:28.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021a90 con 0x7f21c4004620 2021-03-09T10:35:28.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.314+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12879 cs=0 l=1 rev1=1 rx=0x7f21e003e540 tx=0x7f21e007d540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:28.321 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:28.318+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:28.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e001f800 con 0x7f21c4004620 2021-03-09T10:35:28.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:35:28.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0007000 con 0x7f21c4004620 2021-03-09T10:35:28.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.326+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:28.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:28.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:28.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:33.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:33.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:33.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12879 cs=0 l=1 rev1=1 rx=0x7f21e003e540 tx=0x7f21e007d540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:33.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12879 cs=0 l=1 rev1=1 rx=0x7f21e003e540 tx=0x7f21e007d540).stop 2021-03-09T10:35:33.324 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:33.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:33.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:33.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:33.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:33.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095290 con 0x7f21c400a6b0 2021-03-09T10:35:33.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a360 con 0x7f21c40037a0 2021-03-09T10:35:33.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d780 con 0x7f21c4005da0 2021-03-09T10:35:33.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:33.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.318+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:33.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:33.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:33.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:33.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:33.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:33.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:33.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:33.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:35:33.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017300 con 0x7f21c4005da0 2021-03-09T10:35:33.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12889 cs=0 l=1 rev1=1 rx=0x7f21e00370f0 tx=0x7f21e009ae80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:33.331 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:33.326+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44205 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:33.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21c4005da0 2021-03-09T10:35:33.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.326+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4005da0 2021-03-09T10:35:33.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4005da0 2021-03-09T10:35:33.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.330+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:33.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:33.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:33.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:38.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:38.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:38.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12889 cs=0 l=1 rev1=1 rx=0x7f21e00370f0 tx=0x7f21e009ae80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:38.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 secure :-1 s=READY pgs=12889 cs=0 l=1 rev1=1 rx=0x7f21e00370f0 tx=0x7f21e009ae80).stop 2021-03-09T10:35:38.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:38.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=12889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:38.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:38.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:38.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:38.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095290 con 0x7f21c400a6b0 2021-03-09T10:35:38.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029cd0 con 0x7f21c40037a0 2021-03-09T10:35:38.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf60 con 0x7f21c4004620 2021-03-09T10:35:38.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:38.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:38.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:38.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:38.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:38.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:38.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:38.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:38.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:38.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a590 con 0x7f21c4004620 2021-03-09T10:35:38.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.330+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a030 con 0x7f21c4004620 2021-03-09T10:35:38.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12902 cs=0 l=1 rev1=1 rx=0x7f21e4041730 tx=0x7f21e4038ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:38.338 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:38.334+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44243 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:38.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4062350 con 0x7f21c4004620 2021-03-09T10:35:38.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4062350 con 0x7f21c4004620 2021-03-09T10:35:38.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.342+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21c4004620 2021-03-09T10:35:38.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.342+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:38.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.342+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:38.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:38.342+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:43.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.334+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:43.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.334+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:43.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12902 cs=0 l=1 rev1=1 rx=0x7f21e4041730 tx=0x7f21e4038ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:43.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12902 cs=0 l=1 rev1=1 rx=0x7f21e4041730 tx=0x7f21e4038ff0).stop 2021-03-09T10:35:43.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:43.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:43.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:43.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:43.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:43.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095290 con 0x7f21c400a6b0 2021-03-09T10:35:43.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c40037a0 2021-03-09T10:35:43.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:43.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:35:43.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:43.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:43.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:43.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:43.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:43.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:43.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:43.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:43.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017300 con 0x7f21c4005da0 2021-03-09T10:35:43.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4005da0 2021-03-09T10:35:43.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c960 secure :-1 s=READY pgs=12910 cs=0 l=1 rev1=1 rx=0x7f21e0006330 tx=0x7f21e002af60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:43.347 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:43.346+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44281 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:43.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21c4005da0 2021-03-09T10:35:43.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003cda0 con 0x7f21c4005da0 2021-03-09T10:35:43.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0007d60 con 0x7f21c4005da0 2021-03-09T10:35:43.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.354+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:43.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:43.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:43.354+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:48.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:48.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:48.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c960 secure :-1 s=READY pgs=12910 cs=0 l=1 rev1=1 rx=0x7f21e0006330 tx=0x7f21e002af60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:48.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c960 secure :-1 s=READY pgs=12910 cs=0 l=1 rev1=1 rx=0x7f21e0006330 tx=0x7f21e002af60).stop 2021-03-09T10:35:48.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04c960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:48.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c960 unknown :-1 s=CLOSED pgs=12910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:48.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:48.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c400a6b0 2021-03-09T10:35:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:35:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041530 con 0x7f21c4004620 2021-03-09T10:35:48.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:48.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:48.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:48.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:48.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:48.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041670 con 0x7f21c4004620 2021-03-09T10:35:48.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a590 con 0x7f21c4004620 2021-03-09T10:35:48.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.350+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12920 cs=0 l=1 rev1=1 rx=0x7f21cc039bb0 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:48.357 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:48.354+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44319 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:48.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d1c0 con 0x7f21c4004620 2021-03-09T10:35:48.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21c4004620 2021-03-09T10:35:48.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4004620 2021-03-09T10:35:48.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.358+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:48.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:48.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:48.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:53.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:53.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:53.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12920 cs=0 l=1 rev1=1 rx=0x7f21cc039bb0 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:53.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12920 cs=0 l=1 rev1=1 rx=0x7f21cc039bb0 tx=0x7f21e4053ed0).stop 2021-03-09T10:35:53.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:53.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:53.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:53.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c400a6b0 2021-03-09T10:35:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:35:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a590 con 0x7f21c4005da0 2021-03-09T10:35:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:53.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:53.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:53.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:53.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:53.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:53.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:53.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:53.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:53.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041800 con 0x7f21c4005da0 2021-03-09T10:35:53.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017300 con 0x7f21c4005da0 2021-03-09T10:35:53.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.358+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c8f0 secure :-1 s=READY pgs=12931 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0004160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:53.364 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:53.362+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44357 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:53.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0090 con 0x7f21c4005da0 2021-03-09T10:35:53.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0090e80 con 0x7f21c4005da0 2021-03-09T10:35:53.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0007d60 con 0x7f21c4005da0 2021-03-09T10:35:53.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.370+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:53.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:53.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:53.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:35:57.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:57.545 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:35:58.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:58.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:58.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c8f0 secure :-1 s=READY pgs=12931 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0004160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:58.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c8f0 secure :-1 s=READY pgs=12931 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0004160).stop 2021-03-09T10:35:58.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04c8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:58.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04c8f0 unknown :-1 s=CLOSED pgs=12931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:58.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:58.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:58.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:58.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c400a6b0 2021-03-09T10:35:58.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:35:58.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094be0 con 0x7f21c4004620 2021-03-09T10:35:58.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:35:58.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:58.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:35:58.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:58.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.362+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:58.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:58.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:58.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:58.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:58.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094d20 con 0x7f21c4004620 2021-03-09T10:35:58.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0940c0 con 0x7f21c4004620 2021-03-09T10:35:58.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12944 cs=0 l=1 rev1=1 rx=0x7f21dc093010 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:58.372 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:35:58.370+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44395 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:35:58.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00a780 con 0x7f21c4004620 2021-03-09T10:35:58.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc05f110 con 0x7f21c4004620 2021-03-09T10:35:58.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc078630 con 0x7f21c4004620 2021-03-09T10:35:58.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.378+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:35:58.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.378+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:35:58.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:35:58.378+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:03.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:03.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:03.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12944 cs=0 l=1 rev1=1 rx=0x7f21dc093010 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:03.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12944 cs=0 l=1 rev1=1 rx=0x7f21dc093010 tx=0x7f21dc010d90).stop 2021-03-09T10:36:03.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:03.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:03.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:03.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:03.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:03.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094be0 con 0x7f21c400a6b0 2021-03-09T10:36:03.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c40037a0 2021-03-09T10:36:03.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:36:03.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:03.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:03.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:03.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.370+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:03.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:03.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:03.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:03.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:03.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:03.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04cc90 con 0x7f21c4005da0 2021-03-09T10:36:03.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.374+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094d20 con 0x7f21c4005da0 2021-03-09T10:36:03.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.378+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04a030 secure :-1 s=READY pgs=12956 cs=0 l=1 rev1=1 rx=0x7f21dc006040 tx=0x7f21dc0890e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:03.383 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:03.382+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:03.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21c4005da0 2021-03-09T10:36:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4005da0 2021-03-09T10:36:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0108a0 con 0x7f21c4005da0 2021-03-09T10:36:03.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.390+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:03.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:03.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:03.390+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:08.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:08.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:08.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04a030 secure :-1 s=READY pgs=12956 cs=0 l=1 rev1=1 rx=0x7f21dc006040 tx=0x7f21dc0890e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:08.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04a030 secure :-1 s=READY pgs=12956 cs=0 l=1 rev1=1 rx=0x7f21dc006040 tx=0x7f21dc0890e0).stop 2021-03-09T10:36:08.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc04a030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:08.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc04a030 unknown :-1 s=CLOSED pgs=12956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:08.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:08.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:08.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:08.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c400a6b0 2021-03-09T10:36:08.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046b10 con 0x7f21c40037a0 2021-03-09T10:36:08.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030bd0 con 0x7f21c4004620 2021-03-09T10:36:08.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:08.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:08.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:08.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:08.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:08.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:08.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:08.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043ea0 con 0x7f21c4004620 2021-03-09T10:36:08.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.382+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04cc90 con 0x7f21c4004620 2021-03-09T10:36:08.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.386+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12964 cs=0 l=1 rev1=1 rx=0x7f21dc0622e0 tx=0x7f21dc015ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:08.396 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:08.390+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44471 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:08.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.390+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083d10 con 0x7f21c4004620 2021-03-09T10:36:08.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.394+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083d10 con 0x7f21c4004620 2021-03-09T10:36:08.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.398+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:36:08.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.398+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:08.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.398+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:08.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:08.398+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:13.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12964 cs=0 l=1 rev1=1 rx=0x7f21dc0622e0 tx=0x7f21dc015ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:13.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12964 cs=0 l=1 rev1=1 rx=0x7f21dc0622e0 tx=0x7f21dc015ad0).stop 2021-03-09T10:36:13.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:13.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0949a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:13.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036e80 con 0x7f21c400a6b0 2021-03-09T10:36:13.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029e70 con 0x7f21c40037a0 2021-03-09T10:36:13.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0959c0 con 0x7f21c4005da0 2021-03-09T10:36:13.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:13.396 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.389+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0949a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:13.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:13.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d270 con 0x7f21c4005da0 2021-03-09T10:36:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:36:13.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.393+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0949a0 secure :-1 s=READY pgs=12977 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc018f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:13.399 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:13.397+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44508 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:13.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083d10 con 0x7f21c4005da0 2021-03-09T10:36:13.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.401+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083d10 con 0x7f21c4005da0 2021-03-09T10:36:13.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c4005da0 2021-03-09T10:36:13.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.405+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:13.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.405+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:13.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:13.405+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:18.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0949a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:18.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0949a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:18.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0949a0 secure :-1 s=READY pgs=12977 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc018f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:18.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0949a0 secure :-1 s=READY pgs=12977 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc018f60).stop 2021-03-09T10:36:18.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0949a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:18.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0949a0 unknown :-1 s=CLOSED pgs=12977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:18.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:18.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:18.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:18.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036e60 con 0x7f21c400a6b0 2021-03-09T10:36:18.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00cbf0 con 0x7f21c40037a0 2021-03-09T10:36:18.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0953d0 con 0x7f21c4004620 2021-03-09T10:36:18.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:18.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:18.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:18.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:18.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.397+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:18.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:18.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:18.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:18.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:18.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043ea0 con 0x7f21c4004620 2021-03-09T10:36:18.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041800 con 0x7f21c4004620 2021-03-09T10:36:18.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.401+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12986 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e0075bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:18.408 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:18.405+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44547 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:18.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:36:18.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:36:18.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:36:18.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.409+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:18.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:18.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:18.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:23.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:23.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:23.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12986 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e0075bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:23.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=12986 cs=0 l=1 rev1=1 rx=0x7f21e0082010 tx=0x7f21e0075bb0).stop 2021-03-09T10:36:23.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:23.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=12986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:23.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:23.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:23.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:23.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc036e30 con 0x7f21c400a6b0 2021-03-09T10:36:23.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041800 con 0x7f21c40037a0 2021-03-09T10:36:23.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041940 con 0x7f21c4005da0 2021-03-09T10:36:23.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:23.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:23.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:23.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:23.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.405+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:23.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:23.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:23.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:23.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:23.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a030 con 0x7f21c4005da0 2021-03-09T10:36:23.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:36:23.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.409+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094900 secure :-1 s=READY pgs=12999 cs=0 l=1 rev1=1 rx=0x7f21e0046000 tx=0x7f21e003df20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:23.417 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:23.413+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44584 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:23.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21c4005da0 2021-03-09T10:36:23.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21c4005da0 2021-03-09T10:36:23.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21c4005da0 2021-03-09T10:36:23.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.421+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:23.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:23.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:23.421+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:27.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:27.593 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:28.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc094900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:28.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc094900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:28.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094900 secure :-1 s=READY pgs=12999 cs=0 l=1 rev1=1 rx=0x7f21e0046000 tx=0x7f21e003df20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:28.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094900 secure :-1 s=READY pgs=12999 cs=0 l=1 rev1=1 rx=0x7f21e0046000 tx=0x7f21e003df20).stop 2021-03-09T10:36:28.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc094900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:28.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc094900 unknown :-1 s=CLOSED pgs=12999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:28.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:28.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:28.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:28.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c320 con 0x7f21c400a6b0 2021-03-09T10:36:28.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:36:28.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0924d0 con 0x7f21c4004620 2021-03-09T10:36:28.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:28.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:28.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:28.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:28.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:28.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:28.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:28.424 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:28.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:28.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092610 con 0x7f21c4004620 2021-03-09T10:36:28.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.417+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a030 con 0x7f21c4004620 2021-03-09T10:36:28.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.421+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13011 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003c0f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:28.426 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:28.421+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44623 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:28.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0070 con 0x7f21c4004620 2021-03-09T10:36:28.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:36:28.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00ac070 con 0x7f21c4004620 2021-03-09T10:36:28.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.425+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:28.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.429+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:28.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:28.429+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:33.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:33.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:33.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13011 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003c0f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:33.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13011 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003c0f0).stop 2021-03-09T10:36:33.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:33.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:33.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:33.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:33.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:33.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c400a6b0 2021-03-09T10:36:33.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0924d0 con 0x7f21c40037a0 2021-03-09T10:36:33.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:33.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092610 con 0x7f21c4005da0 2021-03-09T10:36:33.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:33.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:33.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:33.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:33.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092750 con 0x7f21c4005da0 2021-03-09T10:36:33.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4005da0 2021-03-09T10:36:33.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.429+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0948b0 secure :-1 s=READY pgs=13018 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:33.434 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:33.429+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44660 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:33.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0080 con 0x7f21c4005da0 2021-03-09T10:36:33.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004070 con 0x7f21c4005da0 2021-03-09T10:36:33.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00559e0 con 0x7f21c4005da0 2021-03-09T10:36:33.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.437+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:33.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.437+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:33.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:33.437+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:38.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:38.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:38.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0948b0 secure :-1 s=READY pgs=13018 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:38.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0948b0 secure :-1 s=READY pgs=13018 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0003820).stop 2021-03-09T10:36:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0948b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0948b0 unknown :-1 s=CLOSED pgs=13018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:38.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.429+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c400a6b0 2021-03-09T10:36:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a010 con 0x7f21c40037a0 2021-03-09T10:36:38.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a150 con 0x7f21c4004620 2021-03-09T10:36:38.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:38.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:38.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:38.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:38.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:38.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:38.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a290 con 0x7f21c4004620 2021-03-09T10:36:38.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041670 con 0x7f21c4004620 2021-03-09T10:36:38.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13028 cs=0 l=1 rev1=1 rx=0x7f21cc043870 tx=0x7f21e40b1120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:38.442 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:38.437+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44699 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:38.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4004620 2021-03-09T10:36:38.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4004620 2021-03-09T10:36:38.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4004620 2021-03-09T10:36:38.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.445+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:38.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.445+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:38.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:38.445+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:43.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:43.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:43.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13028 cs=0 l=1 rev1=1 rx=0x7f21cc043870 tx=0x7f21e40b1120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:43.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13028 cs=0 l=1 rev1=1 rx=0x7f21cc043870 tx=0x7f21e40b1120).stop 2021-03-09T10:36:43.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:43.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:43.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039f40 con 0x7f21c400a6b0 2021-03-09T10:36:43.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:36:43.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:36:43.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:43.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:43.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:43.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:43.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:43.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:43.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:43.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:43.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:43.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d0b0 con 0x7f21c4005da0 2021-03-09T10:36:43.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d270 con 0x7f21c4005da0 2021-03-09T10:36:43.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13038 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0192e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:43.452 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:43.449+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44736 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:43.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:36:43.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.449+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:36:43.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc06aca0 con 0x7f21c4005da0 2021-03-09T10:36:43.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.453+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:43.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.453+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:43.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:43.453+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:48.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:48.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:48.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13038 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0192e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:48.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13038 cs=0 l=1 rev1=1 rx=0x7f21dc000bc0 tx=0x7f21dc0192e0).stop 2021-03-09T10:36:48.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:48.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=CLOSED pgs=13038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:48.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:48.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:48.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:48.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0134e0 con 0x7f21c400a6b0 2021-03-09T10:36:48.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:36:48.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:36:48.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:48.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:48.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:48.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:48.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:48.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:48.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:48.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:48.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:48.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0948b0 con 0x7f21c4004620 2021-03-09T10:36:48.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d270 con 0x7f21c4004620 2021-03-09T10:36:48.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13048 cs=0 l=1 rev1=1 rx=0x7f21dc0ac220 tx=0x7f21dc0abef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:48.461 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:48.457+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44775 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:48.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:36:48.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:36:48.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.461+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05cc20 con 0x7f21c4004620 2021-03-09T10:36:48.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.461+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:48.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.461+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:48.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:48.461+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:53.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:53.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:53.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13048 cs=0 l=1 rev1=1 rx=0x7f21dc0ac220 tx=0x7f21dc0abef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:53.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13048 cs=0 l=1 rev1=1 rx=0x7f21dc0ac220 tx=0x7f21dc0abef0).stop 2021-03-09T10:36:53.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:53.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:53.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:53.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:53.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:53.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c400a6b0 2021-03-09T10:36:53.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d410 con 0x7f21c40037a0 2021-03-09T10:36:53.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:36:53.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:53.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:53.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:53.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:53.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.457+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:53.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:53.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:53.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:53.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:53.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0131a0 con 0x7f21c4005da0 2021-03-09T10:36:53.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0948b0 con 0x7f21c4005da0 2021-03-09T10:36:53.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.461+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13059 cs=0 l=1 rev1=1 rx=0x7f21cc021a00 tx=0x7f21dc029120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:53.471 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:53.469+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44812 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:53.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062ec0 con 0x7f21c4005da0 2021-03-09T10:36:53.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062ce0 con 0x7f21c4005da0 2021-03-09T10:36:53.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0292a0 con 0x7f21c4005da0 2021-03-09T10:36:53.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.469+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.473+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:53.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:53.473+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:36:57.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:57.641 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:36:58.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:36:58.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:58.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13059 cs=0 l=1 rev1=1 rx=0x7f21cc021a00 tx=0x7f21dc029120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:58.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13059 cs=0 l=1 rev1=1 rx=0x7f21cc021a00 tx=0x7f21dc029120).stop 2021-03-09T10:36:58.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:58.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=CLOSED pgs=13059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:58.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:58.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:58.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:58.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057bd0 con 0x7f21c400a6b0 2021-03-09T10:36:58.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:36:58.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0134d0 con 0x7f21c4004620 2021-03-09T10:36:58.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:36:58.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:36:58.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:58.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:58.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:58.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d410 con 0x7f21c4004620 2021-03-09T10:36:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.469+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc092750 con 0x7f21c4004620 2021-03-09T10:36:58.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.473+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13071 cs=0 l=1 rev1=1 rx=0x7f21e0013580 tx=0x7f21e003c080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:58.480 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:36:58.477+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44851 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:36:58.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:36:58.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.481+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:36:58.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.485+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:36:58.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.485+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:36:58.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.485+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:36:58.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:36:58.485+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:03.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13071 cs=0 l=1 rev1=1 rx=0x7f21e0013580 tx=0x7f21e003c080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13071 cs=0 l=1 rev1=1 rx=0x7f21e0013580 tx=0x7f21e003c080).stop 2021-03-09T10:37:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:03.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc03a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c400a6b0 2021-03-09T10:37:03.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c40037a0 2021-03-09T10:37:03.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d0b0 con 0x7f21c4005da0 2021-03-09T10:37:03.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:03.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:03.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:03.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:03.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc03a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:03.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:03.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:03.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0948b0 con 0x7f21c4005da0 2021-03-09T10:37:03.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0131a0 con 0x7f21c4005da0 2021-03-09T10:37:03.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc03a0d0 secure :-1 s=READY pgs=13080 cs=0 l=1 rev1=1 rx=0x7f21dc029820 tx=0x7f21dc018f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:03.489 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:03.484+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44888 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:03.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b8070 con 0x7f21c4005da0 2021-03-09T10:37:03.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0298b0 con 0x7f21c4005da0 2021-03-09T10:37:03.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c4005da0 2021-03-09T10:37:03.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.488+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:03.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:03.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:03.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:08.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:37:08.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:08.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc03a0d0 secure :-1 s=READY pgs=13080 cs=0 l=1 rev1=1 rx=0x7f21dc029820 tx=0x7f21dc018f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:08.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc03a0d0 secure :-1 s=READY pgs=13080 cs=0 l=1 rev1=1 rx=0x7f21dc029820 tx=0x7f21dc018f60).stop 2021-03-09T10:37:08.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc03a0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:08.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc03a0d0 unknown :-1 s=CLOSED pgs=13080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:08.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:08.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:08.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:08.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093f80 con 0x7f21c400a6b0 2021-03-09T10:37:08.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c40037a0 2021-03-09T10:37:08.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4004620 2021-03-09T10:37:08.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:08.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:08.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:08.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:08.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:08.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:08.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:08.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:08.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:08.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094bb0 con 0x7f21c4004620 2021-03-09T10:37:08.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a290 con 0x7f21c4004620 2021-03-09T10:37:08.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.488+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13091 cs=0 l=1 rev1=1 rx=0x7f21e4039060 tx=0x7f21e404f280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:08.498 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:08.492+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44927 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:08.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21c4004620 2021-03-09T10:37:08.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4004620 2021-03-09T10:37:08.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4004620 2021-03-09T10:37:08.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.496+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:08.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.496+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:08.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:08.496+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:13.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:13.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:13.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13091 cs=0 l=1 rev1=1 rx=0x7f21e4039060 tx=0x7f21e404f280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:13.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13091 cs=0 l=1 rev1=1 rx=0x7f21e4039060 tx=0x7f21e404f280).stop 2021-03-09T10:37:13.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:13.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:13.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:13.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:13.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:13.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094100 con 0x7f21c400a6b0 2021-03-09T10:37:13.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c40037a0 2021-03-09T10:37:13.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:37:13.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:13.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:13.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:13.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:13.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:13.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:13.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:13.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:13.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:13.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:37:13.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d410 con 0x7f21c4005da0 2021-03-09T10:37:13.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13105 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:13.506 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:13.500+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44964 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:13.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21c4005da0 2021-03-09T10:37:13.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0090e90 con 0x7f21c4005da0 2021-03-09T10:37:13.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.508+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:37:13.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.508+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:13.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:13.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:13.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:18.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:18.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:18.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13105 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:18.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13105 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007c530).stop 2021-03-09T10:37:18.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:18.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=CLOSED pgs=13105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:18.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:18.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:18.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:18.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c400a6b0 2021-03-09T10:37:18.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c40037a0 2021-03-09T10:37:18.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc089610 con 0x7f21c4004620 2021-03-09T10:37:18.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:18.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:18.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:18.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:18.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:18.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:18.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:18.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:18.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:18.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc043ea0 con 0x7f21c4004620 2021-03-09T10:37:18.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d410 con 0x7f21c4004620 2021-03-09T10:37:18.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.508+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13113 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0020e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:18.515 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:18.512+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45003 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:18.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0037d50 con 0x7f21c4004620 2021-03-09T10:37:18.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0037d50 con 0x7f21c4004620 2021-03-09T10:37:18.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.520+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:37:18.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.520+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:18.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.520+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:18.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:18.520+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:23.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:23.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:23.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13113 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0020e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:23.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13113 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0020e90).stop 2021-03-09T10:37:23.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:23.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:23.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:23.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:23.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:23.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dab0 con 0x7f21c400a6b0 2021-03-09T10:37:23.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c40037a0 2021-03-09T10:37:23.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4005da0 2021-03-09T10:37:23.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:23.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:23.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:23.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:23.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:23.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:23.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:23.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a430 con 0x7f21c4005da0 2021-03-09T10:37:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc043ea0 con 0x7f21c4005da0 2021-03-09T10:37:23.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13123 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e007b740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:23.524 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:23.520+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45040 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:23.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.520+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00b0020 con 0x7f21c4005da0 2021-03-09T10:37:23.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.520+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004070 con 0x7f21c4005da0 2021-03-09T10:37:23.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007bdc0 con 0x7f21c4005da0 2021-03-09T10:37:23.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.524+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:23.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.524+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:23.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:23.524+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:27.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:27.688 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:28.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:28.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13123 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e007b740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13123 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e007b740).stop 2021-03-09T10:37:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:28.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=CLOSED pgs=13123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:28.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01dc50 con 0x7f21c400a6b0 2021-03-09T10:37:28.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c40037a0 2021-03-09T10:37:28.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4004620 2021-03-09T10:37:28.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:28.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:28.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:28.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:28.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:28.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:28.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d410 con 0x7f21c4004620 2021-03-09T10:37:28.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21c4004620 2021-03-09T10:37:28.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13134 cs=0 l=1 rev1=1 rx=0x7f21e0004230 tx=0x7f21e007c5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:28.532 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:28.528+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45079 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:28.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.528+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003e3d0 con 0x7f21c4004620 2021-03-09T10:37:28.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.528+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003e1a0 con 0x7f21c4004620 2021-03-09T10:37:28.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007bdc0 con 0x7f21c4004620 2021-03-09T10:37:28.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.532+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:28.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.532+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:28.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:28.532+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:33.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:37:33.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:33.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13134 cs=0 l=1 rev1=1 rx=0x7f21e0004230 tx=0x7f21e007c5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:33.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13134 cs=0 l=1 rev1=1 rx=0x7f21e0004230 tx=0x7f21e007c5a0).stop 2021-03-09T10:37:33.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:33.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:33.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ddf0 con 0x7f21c400a6b0 2021-03-09T10:37:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c40037a0 2021-03-09T10:37:33.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4005da0 2021-03-09T10:37:33.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:33.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:33.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:33.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:33.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:33.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:33.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:33.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:33.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:33.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:37:33.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094bb0 con 0x7f21c4005da0 2021-03-09T10:37:33.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.532+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13147 cs=0 l=1 rev1=1 rx=0x7f21e40a5510 tx=0x7f21e407d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:33.545 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:33.536+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45116 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:33.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.536+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1190 con 0x7f21c4005da0 2021-03-09T10:37:33.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4005da0 2021-03-09T10:37:33.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1190 con 0x7f21c4005da0 2021-03-09T10:37:33.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.544+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:33.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.544+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:33.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:33.544+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:38.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:38.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:38.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13147 cs=0 l=1 rev1=1 rx=0x7f21e40a5510 tx=0x7f21e407d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:38.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13147 cs=0 l=1 rev1=1 rx=0x7f21e40a5510 tx=0x7f21e407d160).stop 2021-03-09T10:37:38.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:38.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=CLOSED pgs=13147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:38.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:38.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:38.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:38.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092900 con 0x7f21c400a6b0 2021-03-09T10:37:38.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc028470 con 0x7f21c40037a0 2021-03-09T10:37:38.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094a40 con 0x7f21c4004620 2021-03-09T10:37:38.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:38.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:38.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:38.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:38.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:38.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:38.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:38.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:38.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:38.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:37:38.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.540+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0948b0 con 0x7f21c4004620 2021-03-09T10:37:38.550 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13159 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc02dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:38.551 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:38.544+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45155 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:38.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.548+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:37:38.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.552+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0058a0 con 0x7f21c4004620 2021-03-09T10:37:38.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:37:38.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.556+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:38.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.556+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:38.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:38.556+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:43.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.544+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:43.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.544+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:43.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13159 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc02dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:43.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13159 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc02dfc0).stop 2021-03-09T10:37:43.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:43.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:43.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:43.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:43.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:43.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c400a6b0 2021-03-09T10:37:43.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096c50 con 0x7f21c40037a0 2021-03-09T10:37:43.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096d90 con 0x7f21c4005da0 2021-03-09T10:37:43.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:43.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:43.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:43.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:43.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:43.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:43.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:43.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:43.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:43.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc096ed0 con 0x7f21c4005da0 2021-03-09T10:37:43.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0948b0 con 0x7f21c4005da0 2021-03-09T10:37:43.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.548+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13167 cs=0 l=1 rev1=1 rx=0x7f21dc061dd0 tx=0x7f21dc062ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:43.559 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:43.552+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45192 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:43.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005f10 con 0x7f21c4005da0 2021-03-09T10:37:43.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:37:43.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.560+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc05ca10 con 0x7f21c4005da0 2021-03-09T10:37:43.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.560+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:43.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.560+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:43.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:43.560+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:48.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:37:48.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:48.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13167 cs=0 l=1 rev1=1 rx=0x7f21dc061dd0 tx=0x7f21dc062ce0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:48.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 secure :-1 s=READY pgs=13167 cs=0 l=1 rev1=1 rx=0x7f21dc061dd0 tx=0x7f21dc062ce0).stop 2021-03-09T10:37:48.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc041640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:48.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc041640 unknown :-1 s=CLOSED pgs=13167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:48.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:48.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:48.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:48.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0579c0 con 0x7f21c400a6b0 2021-03-09T10:37:48.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057b00 con 0x7f21c40037a0 2021-03-09T10:37:48.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0129e0 con 0x7f21c4004620 2021-03-09T10:37:48.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:48.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:48.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:48.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:48.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:48.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:48.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:48.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:48.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:48.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012b20 con 0x7f21c4004620 2021-03-09T10:37:48.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094bb0 con 0x7f21c4004620 2021-03-09T10:37:48.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13176 cs=0 l=1 rev1=1 rx=0x7f21cc0977a0 tx=0x7f21e40b4970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:48.568 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:48.564+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45231 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:48.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4028350 con 0x7f21c4004620 2021-03-09T10:37:48.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21c4004620 2021-03-09T10:37:48.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21c4004620 2021-03-09T10:37:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.568+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.568+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:48.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:48.568+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:53.569 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:53.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:53.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13176 cs=0 l=1 rev1=1 rx=0x7f21cc0977a0 tx=0x7f21e40b4970).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:53.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13176 cs=0 l=1 rev1=1 rx=0x7f21cc0977a0 tx=0x7f21e40b4970).stop 2021-03-09T10:37:53.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:53.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:53.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:53.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:53.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:53.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0129e0 con 0x7f21c400a6b0 2021-03-09T10:37:53.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a710 con 0x7f21c40037a0 2021-03-09T10:37:53.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:37:53.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:53.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:53.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:53.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:53.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:53.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:53.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:53.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:53.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:53.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012cb0 con 0x7f21c4005da0 2021-03-09T10:37:53.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d410 con 0x7f21c4005da0 2021-03-09T10:37:53.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057820 secure :-1 s=READY pgs=13187 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e002af50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:53.581 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:53.572+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45268 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:53.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000d1c0 con 0x7f21c4005da0 2021-03-09T10:37:53.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.572+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0020e90 con 0x7f21c4005da0 2021-03-09T10:37:53.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4005da0 2021-03-09T10:37:53.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.576+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:53.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:53.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:53.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:37:57.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:57.738 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:37:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057820 secure :-1 s=READY pgs=13187 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e002af50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:58.578 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057820 secure :-1 s=READY pgs=13187 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e002af50).stop 2021-03-09T10:37:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057820 unknown :-1 s=CLOSED pgs=13187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:58.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:58.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:58.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0129e0 con 0x7f21c400a6b0 2021-03-09T10:37:58.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d310 con 0x7f21c40037a0 2021-03-09T10:37:58.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d430 con 0x7f21c4004620 2021-03-09T10:37:58.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:37:58.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:37:58.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:58.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:58.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:37:58.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012cb0 con 0x7f21c4004620 2021-03-09T10:37:58.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.575+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13198 cs=0 l=1 rev1=1 rx=0x7f21e0092fd0 tx=0x7f21e0075c70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:58.585 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:37:58.579+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45307 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:37:58.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0033030 con 0x7f21c4004620 2021-03-09T10:37:58.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.579+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0033190 con 0x7f21c4004620 2021-03-09T10:37:58.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0029110 con 0x7f21c4004620 2021-03-09T10:37:58.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.583+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:37:58.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.583+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:37:58.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:37:58.583+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:03.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:03.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:03.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13198 cs=0 l=1 rev1=1 rx=0x7f21e0092fd0 tx=0x7f21e0075c70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:03.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13198 cs=0 l=1 rev1=1 rx=0x7f21e0092fd0 tx=0x7f21e0075c70).stop 2021-03-09T10:38:03.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:03.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:03.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:03.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:03.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:03.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0129e0 con 0x7f21c400a6b0 2021-03-09T10:38:03.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d310 con 0x7f21c40037a0 2021-03-09T10:38:03.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d410 con 0x7f21c4005da0 2021-03-09T10:38:03.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:03.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:03.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:03.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:03.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:03.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:03.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:03.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:03.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:03.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:38:03.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.583+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b20 con 0x7f21c4005da0 2021-03-09T10:38:03.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.587+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057630 secure :-1 s=READY pgs=13213 cs=0 l=1 rev1=1 rx=0x7f21cc041580 tx=0x7f21e40a6510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:03.595 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:03.591+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45344 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:03.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e403dc40 con 0x7f21c4005da0 2021-03-09T10:38:03.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4005da0 2021-03-09T10:38:03.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e403dc40 con 0x7f21c4005da0 2021-03-09T10:38:03.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.595+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:03.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:03.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:03.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:08.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:08.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:08.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057630 secure :-1 s=READY pgs=13213 cs=0 l=1 rev1=1 rx=0x7f21cc041580 tx=0x7f21e40a6510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:08.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057630 secure :-1 s=READY pgs=13213 cs=0 l=1 rev1=1 rx=0x7f21cc041580 tx=0x7f21e40a6510).stop 2021-03-09T10:38:08.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:08.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057630 unknown :-1 s=CLOSED pgs=13213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:08.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:08.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:08.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:08.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21c400a6b0 2021-03-09T10:38:08.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04d540 con 0x7f21c40037a0 2021-03-09T10:38:08.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237a0 con 0x7f21c4004620 2021-03-09T10:38:08.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:08.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:08.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:08.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:08.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.595+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:08.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:08.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:08.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:08.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:08.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:38:08.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b20 con 0x7f21c4004620 2021-03-09T10:38:08.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13223 cs=0 l=1 rev1=1 rx=0x7f21cc0236d0 tx=0x7f21e40a53f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:08.604 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:08.603+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45382 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:08.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4008da0 con 0x7f21c4004620 2021-03-09T10:38:08.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4004620 2021-03-09T10:38:08.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008da0 con 0x7f21c4004620 2021-03-09T10:38:08.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.607+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:08.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:08.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:08.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:13.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:13.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:13.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13223 cs=0 l=1 rev1=1 rx=0x7f21cc0236d0 tx=0x7f21e40a53f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:13.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13223 cs=0 l=1 rev1=1 rx=0x7f21cc0236d0 tx=0x7f21e40a53f0).stop 2021-03-09T10:38:13.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:13.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:13.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:13.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:13.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc053290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:13.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095860 con 0x7f21c400a6b0 2021-03-09T10:38:13.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21c40037a0 2021-03-09T10:38:13.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:38:13.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:13.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:13.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:13.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:13.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.603+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc053290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:13.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:13.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:13.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:13.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:13.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08da80 con 0x7f21c4005da0 2021-03-09T10:38:13.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc096ed0 con 0x7f21c4005da0 2021-03-09T10:38:13.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.607+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc053290 secure :-1 s=READY pgs=13230 cs=0 l=1 rev1=1 rx=0x7f21dc0961c0 tx=0x7f21dc006000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:13.613 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:13.611+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45420 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:13.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:38:13.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:38:13.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.615+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07edc0 con 0x7f21c4005da0 2021-03-09T10:38:13.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.615+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:13.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:13.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:13.615+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:18.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc053290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:18.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc053290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:18.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc053290 secure :-1 s=READY pgs=13230 cs=0 l=1 rev1=1 rx=0x7f21dc0961c0 tx=0x7f21dc006000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:18.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc053290 secure :-1 s=READY pgs=13230 cs=0 l=1 rev1=1 rx=0x7f21dc0961c0 tx=0x7f21dc006000).stop 2021-03-09T10:38:18.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc053290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:18.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc053290 unknown :-1 s=CLOSED pgs=13230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:18.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:18.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:18.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:18.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224d0 con 0x7f21c400a6b0 2021-03-09T10:38:18.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21c40037a0 2021-03-09T10:38:18.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095500 con 0x7f21c4004620 2021-03-09T10:38:18.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:18.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:18.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:18.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.611+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:18.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:18.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:18.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:18.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:18.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:18.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095670 con 0x7f21c4004620 2021-03-09T10:38:18.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b20 con 0x7f21c4004620 2021-03-09T10:38:18.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.615+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13240 cs=0 l=1 rev1=1 rx=0x7f21cc00eac0 tx=0x7f21e404f1a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:18.622 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:18.619+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45458 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:18.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f360 con 0x7f21c4004620 2021-03-09T10:38:18.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404f360 con 0x7f21c4004620 2021-03-09T10:38:18.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:38:18.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.623+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:18.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.627+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:18.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:18.627+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:23.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:23.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:23.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13240 cs=0 l=1 rev1=1 rx=0x7f21cc00eac0 tx=0x7f21e404f1a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:23.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13240 cs=0 l=1 rev1=1 rx=0x7f21cc00eac0 tx=0x7f21e404f1a0).stop 2021-03-09T10:38:23.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:23.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:23.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:23.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:23.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:23.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:38:23.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095500 con 0x7f21c40037a0 2021-03-09T10:38:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac30 con 0x7f21c4005da0 2021-03-09T10:38:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:23.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.619+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:23.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:23.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012cc0 con 0x7f21c4005da0 2021-03-09T10:38:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08da80 con 0x7f21c4005da0 2021-03-09T10:38:23.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.623+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0573a0 secure :-1 s=READY pgs=13251 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc010dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:23.630 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:23.627+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45496 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:23.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.627+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4005da0 2021-03-09T10:38:23.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.627+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:38:23.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc07edc0 con 0x7f21c4005da0 2021-03-09T10:38:23.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.631+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:23.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.635+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:23.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:23.635+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:27.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:27.784 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:28.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.627+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:28.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.627+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.627+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0573a0 secure :-1 s=READY pgs=13251 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc010dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.627+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0573a0 secure :-1 s=READY pgs=13251 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc010dd0).stop 2021-03-09T10:38:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.627+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc0573a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.627+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc0573a0 unknown :-1 s=CLOSED pgs=13251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:28.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:28.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:28.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095500 con 0x7f21c400a6b0 2021-03-09T10:38:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21c40037a0 2021-03-09T10:38:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c4004620 2021-03-09T10:38:28.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:28.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:28.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:28.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:28.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:28.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:28.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:28.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:28.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:28.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21c4004620 2021-03-09T10:38:28.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095670 con 0x7f21c4004620 2021-03-09T10:38:28.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.635+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13262 cs=0 l=1 rev1=1 rx=0x7f21cc0948e0 tx=0x7f21e4009310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:28.639 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:28.639+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45534 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:28.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21c4004620 2021-03-09T10:38:28.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a5120 con 0x7f21c4004620 2021-03-09T10:38:28.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.643+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4048180 con 0x7f21c4004620 2021-03-09T10:38:28.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.643+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:28.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.643+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:28.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:28.643+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:33.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:33.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:33.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13262 cs=0 l=1 rev1=1 rx=0x7f21cc0948e0 tx=0x7f21e4009310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:33.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13262 cs=0 l=1 rev1=1 rx=0x7f21cc0948e0 tx=0x7f21e4009310).stop 2021-03-09T10:38:33.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:33.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:33.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:33.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:33.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:33.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21c400a6b0 2021-03-09T10:38:33.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039f40 con 0x7f21c40037a0 2021-03-09T10:38:33.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c4005da0 2021-03-09T10:38:33.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:33.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:33.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:33.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:33.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.639+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:33.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:33.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:33.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:33.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:33.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095810 con 0x7f21c4005da0 2021-03-09T10:38:33.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012cc0 con 0x7f21c4005da0 2021-03-09T10:38:33.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.643+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057060 secure :-1 s=READY pgs=13273 cs=0 l=1 rev1=1 rx=0x7f21dc029d90 tx=0x7f21dc0800f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:33.648 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:33.647+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45572 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:33.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b8a40 con 0x7f21c4005da0 2021-03-09T10:38:33.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b8a40 con 0x7f21c4005da0 2021-03-09T10:38:33.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.655+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4005da0 2021-03-09T10:38:33.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.655+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:33.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.655+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:33.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:33.655+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:38.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:38.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:38.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057060 secure :-1 s=READY pgs=13273 cs=0 l=1 rev1=1 rx=0x7f21dc029d90 tx=0x7f21dc0800f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:38.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057060 secure :-1 s=READY pgs=13273 cs=0 l=1 rev1=1 rx=0x7f21dc029d90 tx=0x7f21dc0800f0).stop 2021-03-09T10:38:38.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc057060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:38.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc057060 unknown :-1 s=CLOSED pgs=13273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:38.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:38.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:38.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:38.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039f40 con 0x7f21c400a6b0 2021-03-09T10:38:38.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21c40037a0 2021-03-09T10:38:38.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c4004620 2021-03-09T10:38:38.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:38.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:38.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:38.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:38.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.647+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:38.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:38.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:38.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:38.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:38.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc052f30 con 0x7f21c4004620 2021-03-09T10:38:38.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:38:38.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.651+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13283 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0003620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:38.657 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:38.655+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45610 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:38.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c5a0 con 0x7f21c4004620 2021-03-09T10:38:38.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c5a0 con 0x7f21c4004620 2021-03-09T10:38:38.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:38:38.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.663+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:38.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.663+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:38.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:38.663+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:43.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:43.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:43.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13283 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0003620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:43.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13283 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0003620).stop 2021-03-09T10:38:43.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:43.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:43.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:43.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:43.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:43.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21c400a6b0 2021-03-09T10:38:43.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c40037a0 2021-03-09T10:38:43.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ebf0 con 0x7f21c4005da0 2021-03-09T10:38:43.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:43.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:43.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:43.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:43.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.655+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:43.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:43.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:43.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:43.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:43.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039f40 con 0x7f21c4005da0 2021-03-09T10:38:43.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21c4005da0 2021-03-09T10:38:43.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056d20 secure :-1 s=READY pgs=13294 cs=0 l=1 rev1=1 rx=0x7f21e4008000 tx=0x7f21e401abb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:43.666 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:43.663+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45648 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:43.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21c4005da0 2021-03-09T10:38:43.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.663+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016080 con 0x7f21c4005da0 2021-03-09T10:38:43.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4005da0 2021-03-09T10:38:43.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.671+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:43.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.671+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:43.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:43.671+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:48.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:48.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:48.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056d20 secure :-1 s=READY pgs=13294 cs=0 l=1 rev1=1 rx=0x7f21e4008000 tx=0x7f21e401abb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:48.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056d20 secure :-1 s=READY pgs=13294 cs=0 l=1 rev1=1 rx=0x7f21e4008000 tx=0x7f21e401abb0).stop 2021-03-09T10:38:48.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:48.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056d20 unknown :-1 s=CLOSED pgs=13294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:48.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:48.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:48.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:48.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:38:48.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21c40037a0 2021-03-09T10:38:48.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21c4004620 2021-03-09T10:38:48.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:48.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:48.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:48.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:48.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:48.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:48.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:48.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:48.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:48.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac30 con 0x7f21c4004620 2021-03-09T10:38:48.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.666+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc052f30 con 0x7f21c4004620 2021-03-09T10:38:48.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.670+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13305 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e0013150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:48.675 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:48.674+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45686 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:48.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:38:48.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.678+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000a8e0 con 0x7f21c4004620 2021-03-09T10:38:48.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:38:48.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.682+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:48.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.682+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:48.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:48.682+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:53.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:53.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:53.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13305 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e0013150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:53.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13305 cs=0 l=1 rev1=1 rx=0x7f21e002b000 tx=0x7f21e0013150).stop 2021-03-09T10:38:53.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:53.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:53.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:53.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:53.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:53.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:38:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21c40037a0 2021-03-09T10:38:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088f20 con 0x7f21c4005da0 2021-03-09T10:38:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:53.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:53.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:53.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.674+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:53.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:53.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:38:53.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039f40 con 0x7f21c4005da0 2021-03-09T10:38:53.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.678+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056b20 secure :-1 s=READY pgs=13317 cs=0 l=1 rev1=1 rx=0x7f21e4036680 tx=0x7f21e40a5fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:53.684 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:53.682+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45724 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:53.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4036380 con 0x7f21c4005da0 2021-03-09T10:38:53.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.686+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4005da0 2021-03-09T10:38:53.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.686+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4036380 con 0x7f21c4005da0 2021-03-09T10:38:53.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.686+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:53.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.686+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:53.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:53.686+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:38:57.845 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:57.857 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:38:58.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056b20 secure :-1 s=READY pgs=13317 cs=0 l=1 rev1=1 rx=0x7f21e4036680 tx=0x7f21e40a5fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:58.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056b20 secure :-1 s=READY pgs=13317 cs=0 l=1 rev1=1 rx=0x7f21e4036680 tx=0x7f21e40a5fc0).stop 2021-03-09T10:38:58.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:58.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056b20 unknown :-1 s=CLOSED pgs=13317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:58.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:58.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:58.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:58.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:38:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d580 con 0x7f21c40037a0 2021-03-09T10:38:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c510 con 0x7f21c4004620 2021-03-09T10:38:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:38:58.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:58.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:38:58.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:58.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.682+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:58.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:58.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:58.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:58.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:58.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:38:58.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095810 con 0x7f21c4004620 2021-03-09T10:38:58.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.686+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13328 cs=0 l=1 rev1=1 rx=0x7f21cc094400 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:58.693 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:38:58.690+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45762 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:38:58.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc062ec0 con 0x7f21c4004620 2021-03-09T10:38:58.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.690+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b8a40 con 0x7f21c4004620 2021-03-09T10:38:58.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.698+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4004620 2021-03-09T10:38:58.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.698+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:38:58.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.698+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:38:58.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:38:58.698+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:03.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:39:03.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:03.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13328 cs=0 l=1 rev1=1 rx=0x7f21cc094400 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:03.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13328 cs=0 l=1 rev1=1 rx=0x7f21cc094400 tx=0x7f21dc02df50).stop 2021-03-09T10:39:03.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:03.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:03.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:03.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:03.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:03.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:39:03.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0324b0 con 0x7f21c40037a0 2021-03-09T10:39:03.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e040 con 0x7f21c4005da0 2021-03-09T10:39:03.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:03.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:03.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:03.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:03.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:03.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:03.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:03.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:03.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:03.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:39:03.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039f40 con 0x7f21c4005da0 2021-03-09T10:39:03.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.694+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056930 secure :-1 s=READY pgs=13337 cs=0 l=1 rev1=1 rx=0x7f21e409c230 tx=0x7f21e4055420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:03.702 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:03.698+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45800 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:03.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4005da0 2021-03-09T10:39:03.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009e30 con 0x7f21c4005da0 2021-03-09T10:39:03.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4005da0 2021-03-09T10:39:03.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.702+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:03.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.706+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:03.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:03.706+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:08.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:08.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:08.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056930 secure :-1 s=READY pgs=13337 cs=0 l=1 rev1=1 rx=0x7f21e409c230 tx=0x7f21e4055420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:08.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056930 secure :-1 s=READY pgs=13337 cs=0 l=1 rev1=1 rx=0x7f21e409c230 tx=0x7f21e4055420).stop 2021-03-09T10:39:08.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc056930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:08.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc056930 unknown :-1 s=CLOSED pgs=13337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:08.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:08.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:08.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:08.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e1e0 con 0x7f21c400a6b0 2021-03-09T10:39:08.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ae10 con 0x7f21c40037a0 2021-03-09T10:39:08.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0324b0 con 0x7f21c4004620 2021-03-09T10:39:08.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:08.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:08.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:08.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:08.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:08.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:08.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:08.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:08.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:08.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:39:08.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21c4004620 2021-03-09T10:39:08.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13347 cs=0 l=1 rev1=1 rx=0x7f21e000a810 tx=0x7f21e00037b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:08.711 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:08.710+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45838 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:08.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:39:08.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003d850 con 0x7f21c4004620 2021-03-09T10:39:08.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:39:08.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.714+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:08.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.714+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:08.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:08.714+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:13.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:13.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:13.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13347 cs=0 l=1 rev1=1 rx=0x7f21e000a810 tx=0x7f21e00037b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:13.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13347 cs=0 l=1 rev1=1 rx=0x7f21e000a810 tx=0x7f21e00037b0).stop 2021-03-09T10:39:13.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:13.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:13.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:13.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:13.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:13.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e380 con 0x7f21c400a6b0 2021-03-09T10:39:13.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043910 con 0x7f21c40037a0 2021-03-09T10:39:13.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c4005da0 2021-03-09T10:39:13.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:13.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:13.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:13.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:13.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:13.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:13.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:13.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:13.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:13.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0324b0 con 0x7f21c4005da0 2021-03-09T10:39:13.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095810 con 0x7f21c4005da0 2021-03-09T10:39:13.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.714+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13363 cs=0 l=1 rev1=1 rx=0x7f21dc0abe00 tx=0x7f21dc08a240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:13.720 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:13.718+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45876 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:13.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0abe70 con 0x7f21c4005da0 2021-03-09T10:39:13.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4005da0 2021-03-09T10:39:13.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4005da0 2021-03-09T10:39:13.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.726+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:13.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.726+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:13.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:13.726+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:18.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:18.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:18.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13363 cs=0 l=1 rev1=1 rx=0x7f21dc0abe00 tx=0x7f21dc08a240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:18.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13363 cs=0 l=1 rev1=1 rx=0x7f21dc0abe00 tx=0x7f21dc08a240).stop 2021-03-09T10:39:18.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:18.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=13363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:18.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:18.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:18.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:18.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e380 con 0x7f21c400a6b0 2021-03-09T10:39:18.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c40037a0 2021-03-09T10:39:18.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc088f60 con 0x7f21c4004620 2021-03-09T10:39:18.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:18.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:18.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:18.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:18.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.718+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:18.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:18.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:18.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:18.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:18.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04b020 con 0x7f21c4004620 2021-03-09T10:39:18.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039f40 con 0x7f21c4004620 2021-03-09T10:39:18.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13368 cs=0 l=1 rev1=1 rx=0x7f21e4007320 tx=0x7f21e4007220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:18.729 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:18.726+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45915 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:18.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4004620 2021-03-09T10:39:18.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009e30 con 0x7f21c4004620 2021-03-09T10:39:18.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4004620 2021-03-09T10:39:18.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.730+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:18.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:18.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:18.730+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:23.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:23.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:23.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13368 cs=0 l=1 rev1=1 rx=0x7f21e4007320 tx=0x7f21e4007220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:23.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13368 cs=0 l=1 rev1=1 rx=0x7f21e4007320 tx=0x7f21e4007220).stop 2021-03-09T10:39:23.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:23.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:23.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:23.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:23.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:23.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b2f0 con 0x7f21c400a6b0 2021-03-09T10:39:23.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e380 con 0x7f21c40037a0 2021-03-09T10:39:23.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c4005da0 2021-03-09T10:39:23.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:23.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.726+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:23.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:23.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:23.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:23.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095810 con 0x7f21c4005da0 2021-03-09T10:39:23.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04b020 con 0x7f21c4005da0 2021-03-09T10:39:23.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.734+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13380 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e407d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:23.739 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:23.734+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45952 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:23.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f8e0 con 0x7f21c4005da0 2021-03-09T10:39:23.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e404f8e0 con 0x7f21c4005da0 2021-03-09T10:39:23.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.746+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008cd0 con 0x7f21c4005da0 2021-03-09T10:39:23.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.746+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:23.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:23.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:23.746+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:27.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:27.907 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:28.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:39:28.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13380 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e407d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13380 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e407d160).stop 2021-03-09T10:39:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:28.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=13380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:28.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c400a6b0 2021-03-09T10:39:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0129e0 con 0x7f21c40037a0 2021-03-09T10:39:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c4004620 2021-03-09T10:39:28.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:28.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:28.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:28.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:28.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.738+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:28.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:28.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:28.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:28.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:28.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b2f0 con 0x7f21c4004620 2021-03-09T10:39:28.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac30 con 0x7f21c4004620 2021-03-09T10:39:28.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.742+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13394 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0075c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:28.749 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:28.746+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45991 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:28.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d2a0 con 0x7f21c4004620 2021-03-09T10:39:28.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003990 con 0x7f21c4004620 2021-03-09T10:39:28.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.754+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:39:28.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.754+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:28.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:28.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:28.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:33.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:33.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:33.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13394 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0075c90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:33.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=13394 cs=0 l=1 rev1=1 rx=0x7f21e00ada60 tx=0x7f21e0075c90).stop 2021-03-09T10:39:33.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:33.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=13394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:33.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:33.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:33.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:33.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026340 con 0x7f21c400a6b0 2021-03-09T10:39:33.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c40037a0 2021-03-09T10:39:33.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ae0 con 0x7f21c4005da0 2021-03-09T10:39:33.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:33.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:33.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:33.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:33.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:33.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:33.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:33.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:33.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:33.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4005da0 2021-03-09T10:39:33.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b2f0 con 0x7f21c4005da0 2021-03-09T10:39:33.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.754+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13402 cs=0 l=1 rev1=1 rx=0x7f21e0090ac0 tx=0x7f21e00907b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:33.758 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:33.754+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46028 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:33.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006b30 con 0x7f21c4005da0 2021-03-09T10:39:33.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.758+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0091ae0 con 0x7f21c4005da0 2021-03-09T10:39:33.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091cd0 con 0x7f21c4005da0 2021-03-09T10:39:33.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.762+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:33.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:33.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:33.762+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:38.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:38.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.753+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:38.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13402 cs=0 l=1 rev1=1 rx=0x7f21e0090ac0 tx=0x7f21e00907b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:38.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.753+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 secure :-1 s=READY pgs=13402 cs=0 l=1 rev1=1 rx=0x7f21e0090ac0 tx=0x7f21e00907b0).stop 2021-03-09T10:39:38.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:38.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=13402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:38.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:38.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:38.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0232c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:38.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041930 con 0x7f21c400a6b0 2021-03-09T10:39:38.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21c40037a0 2021-03-09T10:39:38.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c4004620 2021-03-09T10:39:38.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:38.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:38.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:38.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:38.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:38.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:38.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:38.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:38.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:38.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01b2f0 con 0x7f21c4004620 2021-03-09T10:39:38.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.757+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0324b0 con 0x7f21c4004620 2021-03-09T10:39:38.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.761+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0232c0 secure :-1 s=READY pgs=13412 cs=0 l=1 rev1=1 rx=0x7f21dc08a2b0 tx=0x7f21dc062120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:38.769 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:38.765+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46067 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:38.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:39:38.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.769+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:39:38.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.773+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0aba10 con 0x7f21c4004620 2021-03-09T10:39:38.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.773+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:38.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.773+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:38.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:38.773+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:43.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0232c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:43.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0232c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:43.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0232c0 secure :-1 s=READY pgs=13412 cs=0 l=1 rev1=1 rx=0x7f21dc08a2b0 tx=0x7f21dc062120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:43.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0232c0 secure :-1 s=READY pgs=13412 cs=0 l=1 rev1=1 rx=0x7f21dc08a2b0 tx=0x7f21dc062120).stop 2021-03-09T10:39:43.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0232c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:43.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0232c0 unknown :-1 s=CLOSED pgs=13412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:43.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:43.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc012500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:43.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:43.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c400a6b0 2021-03-09T10:39:43.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237b0 con 0x7f21c40037a0 2021-03-09T10:39:43.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041930 con 0x7f21c4005da0 2021-03-09T10:39:43.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:43.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.765+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:43.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:43.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:43.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc012500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:43.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:43.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:43.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:43.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:43.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0129f0 con 0x7f21c4005da0 2021-03-09T10:39:43.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095810 con 0x7f21c4005da0 2021-03-09T10:39:43.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.769+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc012500 secure :-1 s=READY pgs=13422 cs=0 l=1 rev1=1 rx=0x7f21e40335b0 tx=0x7f21e4055260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:43.778 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:43.773+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46104 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:43.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409b000 con 0x7f21c4005da0 2021-03-09T10:39:43.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4005da0 2021-03-09T10:39:43.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.781+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409b000 con 0x7f21c4005da0 2021-03-09T10:39:43.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.781+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:43.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.781+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:43.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:43.781+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:48.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc012500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:48.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc012500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:48.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc012500 secure :-1 s=READY pgs=13422 cs=0 l=1 rev1=1 rx=0x7f21e40335b0 tx=0x7f21e4055260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:48.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc012500 secure :-1 s=READY pgs=13422 cs=0 l=1 rev1=1 rx=0x7f21e40335b0 tx=0x7f21e4055260).stop 2021-03-09T10:39:48.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 msgr2=0x7f21cc012500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:48.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4005da0 0x7f21cc012500 unknown :-1 s=CLOSED pgs=13422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:48.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc023220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:48.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:48.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:48.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c400a6b0 2021-03-09T10:39:48.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012d20 con 0x7f21c40037a0 2021-03-09T10:39:48.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a800 con 0x7f21c4004620 2021-03-09T10:39:48.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:48.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:48.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:48.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:48.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc023220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:48.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:48.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:48.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:48.785 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:48.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095810 con 0x7f21c4004620 2021-03-09T10:39:48.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.777+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:39:48.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.781+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc023220 secure :-1 s=READY pgs=13432 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:48.787 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:48.785+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46143 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:48.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e001fea0 con 0x7f21c4004620 2021-03-09T10:39:48.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.789+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e001fea0 con 0x7f21c4004620 2021-03-09T10:39:48.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091a10 con 0x7f21c4004620 2021-03-09T10:39:48.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.793+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:48.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:48.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:48.793+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:51.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900002305) for more than 900000000 us, fault. 2021-03-09T10:39:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=10923 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc056ec0).stop 2021-03-09T10:39:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21cc0138f0 2021-03-09T10:39:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T10:39:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=10923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 0x7f21cc012490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:51.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:51.993+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 0x7f21cc012490 secure :-1 s=READY pgs=12731 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc056ec0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:53.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc023220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:53.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc023220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:53.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc023220 secure :-1 s=READY pgs=13432 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:53.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc023220 secure :-1 s=READY pgs=13432 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b3e0).stop 2021-03-09T10:39:53.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc023220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:53.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc023220 unknown :-1 s=CLOSED pgs=13432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:53.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:53.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:53.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:53.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c400a6b0 2021-03-09T10:39:53.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012d20 con 0x7f21c40037a0 2021-03-09T10:39:53.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041930 con 0x7f21cc0138f0 2021-03-09T10:39:53.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:53.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:53.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:53.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:53.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:53.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04a800 con 0x7f21cc0138f0 2021-03-09T10:39:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.785+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0129f0 con 0x7f21cc0138f0 2021-03-09T10:39:53.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.789+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13443 cs=0 l=1 rev1=1 rx=0x7f21cc056ea0 tx=0x7f21e40b0350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:53.796 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:53.793+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46180 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:53.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b0140 con 0x7f21cc0138f0 2021-03-09T10:39:53.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409b000 con 0x7f21cc0138f0 2021-03-09T10:39:53.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.797+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409b000 con 0x7f21cc0138f0 2021-03-09T10:39:53.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.797+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:53.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.797+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:53.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:53.797+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:39:57.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:57.951 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:39:58.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:58.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:58.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13443 cs=0 l=1 rev1=1 rx=0x7f21cc056ea0 tx=0x7f21e40b0350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:58.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13443 cs=0 l=1 rev1=1 rx=0x7f21cc056ea0 tx=0x7f21e40b0350).stop 2021-03-09T10:39:58.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:58.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:58.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc022ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:58.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:58.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:58.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030fc0 con 0x7f21c400a6b0 2021-03-09T10:39:58.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f0a0 con 0x7f21c40037a0 2021-03-09T10:39:58.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041930 con 0x7f21c4004620 2021-03-09T10:39:58.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:39:58.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:58.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:39:58.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:58.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.793+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc022ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:58.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:58.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:58.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:58.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:58.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:39:58.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095810 con 0x7f21c4004620 2021-03-09T10:39:58.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.797+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc022ff0 secure :-1 s=READY pgs=13456 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e00450e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:58.806 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:39:58.801+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46219 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:39:58.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.801+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:39:58.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.801+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:39:58.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:39:58.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.805+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:39:58.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.805+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:39:58.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:39:58.805+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:03.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:40:03.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc022ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:03.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc022ff0 secure :-1 s=READY pgs=13456 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e00450e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:03.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc022ff0 secure :-1 s=READY pgs=13456 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e00450e0).stop 2021-03-09T10:40:03.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc022ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:03.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc022ff0 unknown :-1 s=CLOSED pgs=13456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:03.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:03.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:03.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:03.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:40:03.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a4b0 con 0x7f21c40037a0 2021-03-09T10:40:03.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ce60 con 0x7f21cc0138f0 2021-03-09T10:40:03.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:03.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:03.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:03.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:03.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:03.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:03.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:03.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:03.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:03.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:40:03.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc0138f0 2021-03-09T10:40:03.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13466 cs=0 l=1 rev1=1 rx=0x7f21e0083550 tx=0x7f21e00291f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:03.814 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:03.809+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46255 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:03.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.809+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006b30 con 0x7f21cc0138f0 2021-03-09T10:40:03.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.817+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006b30 con 0x7f21cc0138f0 2021-03-09T10:40:03.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.817+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091cd0 con 0x7f21cc0138f0 2021-03-09T10:40:03.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.817+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:03.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.817+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:03.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:03.817+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:08.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:08.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:08.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13466 cs=0 l=1 rev1=1 rx=0x7f21e0083550 tx=0x7f21e00291f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:08.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13466 cs=0 l=1 rev1=1 rx=0x7f21e0083550 tx=0x7f21e00291f0).stop 2021-03-09T10:40:08.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:08.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:08.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:08.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:08.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:08.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:40:08.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21c40037a0 2021-03-09T10:40:08.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094680 con 0x7f21c4004620 2021-03-09T10:40:08.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:08.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:08.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:08.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:08.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:08.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:08.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:08.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:08.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:08.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc095810 con 0x7f21c4004620 2021-03-09T10:40:08.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:40:08.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.817+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 secure :-1 s=READY pgs=13475 cs=0 l=1 rev1=1 rx=0x7f21e0004fd0 tx=0x7f21e0006b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:08.823 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:08.817+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46295 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:08.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0004e10 con 0x7f21c4004620 2021-03-09T10:40:08.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004e10 con 0x7f21c4004620 2021-03-09T10:40:08.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0015160 con 0x7f21c4004620 2021-03-09T10:40:08.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.821+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:08.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.825+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:08.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:08.825+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:13.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:40:13.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:13.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 secure :-1 s=READY pgs=13475 cs=0 l=1 rev1=1 rx=0x7f21e0004fd0 tx=0x7f21e0006b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:13.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 secure :-1 s=READY pgs=13475 cs=0 l=1 rev1=1 rx=0x7f21e0004fd0 tx=0x7f21e0006b30).stop 2021-03-09T10:40:13.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:13.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 unknown :-1 s=CLOSED pgs=13475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:13.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:13.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:13.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:13.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c400a6b0 2021-03-09T10:40:13.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c40037a0 2021-03-09T10:40:13.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc058ac0 con 0x7f21cc0138f0 2021-03-09T10:40:13.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:13.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:13.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:13.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:13.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:13.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:13.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.821+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:13.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.825+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:13.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.825+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:13.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.825+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:40:13.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.825+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc095810 con 0x7f21cc0138f0 2021-03-09T10:40:13.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.825+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13488 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0051fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:13.832 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:13.829+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46331 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:13.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.829+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00ad0b0 con 0x7f21cc0138f0 2021-03-09T10:40:13.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.833+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00ad0b0 con 0x7f21cc0138f0 2021-03-09T10:40:13.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00370b0 con 0x7f21cc0138f0 2021-03-09T10:40:13.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.837+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:13.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.837+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:13.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:13.837+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:18.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:40:18.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:18.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13488 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0051fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:18.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13488 cs=0 l=1 rev1=1 rx=0x7f21e000b0b0 tx=0x7f21e0051fb0).stop 2021-03-09T10:40:18.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:18.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:18.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:18.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:18.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:18.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c400a6b0 2021-03-09T10:40:18.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021d40 con 0x7f21c40037a0 2021-03-09T10:40:18.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095810 con 0x7f21c4004620 2021-03-09T10:40:18.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:18.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:18.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:18.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:18.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:18.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:18.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:18.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:18.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:18.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:40:18.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:40:18.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.833+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 secure :-1 s=READY pgs=13496 cs=0 l=1 rev1=1 rx=0x7f21e0052040 tx=0x7f21e00a6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:18.841 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:18.837+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46371 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:18.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003e1a0 con 0x7f21c4004620 2021-03-09T10:40:18.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002a100 con 0x7f21c4004620 2021-03-09T10:40:18.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00370b0 con 0x7f21c4004620 2021-03-09T10:40:18.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.841+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:18.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.841+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:18.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:18.841+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:23.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:23.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:23.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 secure :-1 s=READY pgs=13496 cs=0 l=1 rev1=1 rx=0x7f21e0052040 tx=0x7f21e00a6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:23.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 secure :-1 s=READY pgs=13496 cs=0 l=1 rev1=1 rx=0x7f21e0052040 tx=0x7f21e00a6200).stop 2021-03-09T10:40:23.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:23.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa20 unknown :-1 s=CLOSED pgs=13496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:23.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:23.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:23.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:23.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0431f0 con 0x7f21c400a6b0 2021-03-09T10:40:23.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c40037a0 2021-03-09T10:40:23.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc0138f0 2021-03-09T10:40:23.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:23.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:23.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:23.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:23.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:23.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:23.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:23.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:23.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:23.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025fc0 con 0x7f21cc0138f0 2021-03-09T10:40:23.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b2f0 con 0x7f21cc0138f0 2021-03-09T10:40:23.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.841+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13509 cs=0 l=1 rev1=1 rx=0x7f21dc051c10 tx=0x7f21dc005000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:23.855 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:23.849+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46407 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:23.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:40:23.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:40:23.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:40:23.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.857+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:23.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.857+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:23.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:23.857+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:27.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:28.001 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:28.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:28.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:28.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13509 cs=0 l=1 rev1=1 rx=0x7f21dc051c10 tx=0x7f21dc005000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:28.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13509 cs=0 l=1 rev1=1 rx=0x7f21dc051c10 tx=0x7f21dc005000).stop 2021-03-09T10:40:28.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:28.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:28.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:28.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:28.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:28.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c400a6b0 2021-03-09T10:40:28.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fc0 con 0x7f21c40037a0 2021-03-09T10:40:28.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:40:28.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:28.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:28.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:28.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:28.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:28.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022cc0 con 0x7f21c4004620 2021-03-09T10:40:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04a800 con 0x7f21c4004620 2021-03-09T10:40:28.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.853+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa00 secure :-1 s=READY pgs=13519 cs=0 l=1 rev1=1 rx=0x7f21e407d130 tx=0x7f21e4016080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:28.860 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:28.857+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46447 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:28.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21c4004620 2021-03-09T10:40:28.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.865+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21c4004620 2021-03-09T10:40:28.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.865+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21c4004620 2021-03-09T10:40:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.865+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.865+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:28.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:28.865+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:33.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa00 secure :-1 s=READY pgs=13519 cs=0 l=1 rev1=1 rx=0x7f21e407d130 tx=0x7f21e4016080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:33.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa00 secure :-1 s=READY pgs=13519 cs=0 l=1 rev1=1 rx=0x7f21e407d130 tx=0x7f21e4016080).stop 2021-03-09T10:40:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00aa00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:33.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00aa00 unknown :-1 s=CLOSED pgs=13519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:33.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:33.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:33.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:33.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c7d0 con 0x7f21c400a6b0 2021-03-09T10:40:33.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:40:33.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21cc0138f0 2021-03-09T10:40:33.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:33.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:33.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:33.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:33.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:33.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:33.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:33.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025fc0 con 0x7f21cc0138f0 2021-03-09T10:40:33.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.860+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:40:33.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.864+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13532 cs=0 l=1 rev1=1 rx=0x7f21e00ad290 tx=0x7f21e0075bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:33.870 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:33.864+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46483 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:33.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.868+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0004e10 con 0x7f21cc0138f0 2021-03-09T10:40:33.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.872+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004c30 con 0x7f21cc0138f0 2021-03-09T10:40:33.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.872+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:40:33.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.872+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:33.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.876+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:33.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:33.876+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:38.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:38.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:38.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13532 cs=0 l=1 rev1=1 rx=0x7f21e00ad290 tx=0x7f21e0075bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:38.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13532 cs=0 l=1 rev1=1 rx=0x7f21e00ad290 tx=0x7f21e0075bb0).stop 2021-03-09T10:40:38.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:38.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:38.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:38.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:38.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.864+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:38.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:40:38.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c40037a0 2021-03-09T10:40:38.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fc0 con 0x7f21c4004620 2021-03-09T10:40:38.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:38.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:38.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:38.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:38.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:38.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:38.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:38.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:38.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:38.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022e50 con 0x7f21c4004620 2021-03-09T10:40:38.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01b2f0 con 0x7f21c4004620 2021-03-09T10:40:38.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.868+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a960 secure :-1 s=READY pgs=13540 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc077e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:38.879 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:38.876+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46523 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:38.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:40:38.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081160 con 0x7f21c4004620 2021-03-09T10:40:38.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:40:38.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.876+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:38.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.880+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:38.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:38.880+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:43.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:43.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:43.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a960 secure :-1 s=READY pgs=13540 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc077e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:43.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a960 secure :-1 s=READY pgs=13540 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc077e90).stop 2021-03-09T10:40:43.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:43.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a960 unknown :-1 s=CLOSED pgs=13540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:43.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:43.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:43.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:43.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057a80 con 0x7f21c400a6b0 2021-03-09T10:40:43.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:40:43.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21cc0138f0 2021-03-09T10:40:43.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:43.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:43.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:43.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:43.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:43.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:43.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.876+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:43.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.880+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:43.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.880+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:43.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.880+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025fc0 con 0x7f21cc0138f0 2021-03-09T10:40:43.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.880+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022e50 con 0x7f21cc0138f0 2021-03-09T10:40:43.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.880+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13551 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc019280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:43.888 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:43.884+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46559 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:43.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.884+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc055810 con 0x7f21cc0138f0 2021-03-09T10:40:43.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.884+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc055810 con 0x7f21cc0138f0 2021-03-09T10:40:43.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.888+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083c90 con 0x7f21cc0138f0 2021-03-09T10:40:43.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.888+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:43.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.888+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:43.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:43.888+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:48.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:40:48.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:48.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13551 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc019280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:48.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13551 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc019280).stop 2021-03-09T10:40:48.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:48.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:48.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:48.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:48.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:48.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:48.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:40:48.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:48.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c40037a0 2021-03-09T10:40:48.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fc0 con 0x7f21c4004620 2021-03-09T10:40:48.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:48.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:48.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:48.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:48.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:48.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:48.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:48.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac60 con 0x7f21c4004620 2021-03-09T10:40:48.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022cc0 con 0x7f21c4004620 2021-03-09T10:40:48.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.888+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a770 secure :-1 s=READY pgs=13560 cs=0 l=1 rev1=1 rx=0x7f21cc052010 tx=0x7f21e402d8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:48.898 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:48.892+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46599 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:48.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.892+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21c4004620 2021-03-09T10:40:48.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.896+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21c4004620 2021-03-09T10:40:48.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.900+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404fbe0 con 0x7f21c4004620 2021-03-09T10:40:48.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.900+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:48.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.900+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:48.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:48.900+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:53.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:53.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:53.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a770 secure :-1 s=READY pgs=13560 cs=0 l=1 rev1=1 rx=0x7f21cc052010 tx=0x7f21e402d8a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:53.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a770 secure :-1 s=READY pgs=13560 cs=0 l=1 rev1=1 rx=0x7f21cc052010 tx=0x7f21e402d8a0).stop 2021-03-09T10:40:53.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:53.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a770 unknown :-1 s=CLOSED pgs=13560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:53.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:53.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:53.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:53.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:40:53.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c40037a0 2021-03-09T10:40:53.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025fc0 con 0x7f21cc0138f0 2021-03-09T10:40:53.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:53.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:53.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:53.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:53.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:53.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:53.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:53.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:53.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:53.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00adf0 con 0x7f21cc0138f0 2021-03-09T10:40:53.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.896+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:40:53.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.900+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13571 cs=0 l=1 rev1=1 rx=0x7f21e0005450 tx=0x7f21e000b490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:53.907 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:53.904+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46635 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:53.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.904+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:40:53.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.908+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:40:53.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.908+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:40:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.908+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.912+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:53.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:53.912+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:40:58.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:58.066 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:40:58.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:58.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:58.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13571 cs=0 l=1 rev1=1 rx=0x7f21e0005450 tx=0x7f21e000b490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:58.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 secure :-1 s=READY pgs=13571 cs=0 l=1 rev1=1 rx=0x7f21e0005450 tx=0x7f21e000b490).stop 2021-03-09T10:40:58.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc025ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:58.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc025ad0 unknown :-1 s=CLOSED pgs=13571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:58.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:58.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:58.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:58.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:40:58.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:58.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:40:58.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:40:58.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f2e0 con 0x7f21c40037a0 2021-03-09T10:40:58.912 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:58.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ee0 con 0x7f21c4004620 2021-03-09T10:40:58.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:58.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:58.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:58.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:58.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:58.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:40:58.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022e50 con 0x7f21c4004620 2021-03-09T10:40:58.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.908+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a430 secure :-1 s=READY pgs=13582 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc018150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:58.918 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:40:58.912+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46675 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:40:58.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.912+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc021910 con 0x7f21c4004620 2021-03-09T10:40:58.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.912+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:40:58.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.916+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:40:58.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.916+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:40:58.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.916+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:40:58.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:40:58.916+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:03.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:41:03.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:03.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a430 secure :-1 s=READY pgs=13582 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc018150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:03.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a430 secure :-1 s=READY pgs=13582 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc018150).stop 2021-03-09T10:41:03.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:03.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a430 unknown :-1 s=CLOSED pgs=13582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:03.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc023330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:41:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029d70 con 0x7f21c40037a0 2021-03-09T10:41:03.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ab00 con 0x7f21cc0138f0 2021-03-09T10:41:03.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:03.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:03.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc023330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:03.923 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:03.924 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:03.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:03.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012ee0 con 0x7f21cc0138f0 2021-03-09T10:41:03.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00adf0 con 0x7f21cc0138f0 2021-03-09T10:41:03.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.916+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc023330 secure :-1 s=READY pgs=13592 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:03.928 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:03.924+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46711 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:03.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.924+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000a8f0 con 0x7f21cc0138f0 2021-03-09T10:41:03.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000a8f0 con 0x7f21cc0138f0 2021-03-09T10:41:03.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0091d50 con 0x7f21cc0138f0 2021-03-09T10:41:03.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.932+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:03.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.932+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:03.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:03.932+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:08.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc023330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc023330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc023330 secure :-1 s=READY pgs=13592 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc023330 secure :-1 s=READY pgs=13592 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e0003740).stop 2021-03-09T10:41:08.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc023330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc023330 unknown :-1 s=CLOSED pgs=13592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:08.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:08.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029d70 con 0x7f21c400a6b0 2021-03-09T10:41:08.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:08.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f620 con 0x7f21c40037a0 2021-03-09T10:41:08.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00af90 con 0x7f21c4004620 2021-03-09T10:41:08.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:08.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:08.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:08.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:08.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:08.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:41:08.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:41:08.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.928+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a240 secure :-1 s=READY pgs=13608 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc00adc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:08.937 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:08.932+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46751 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:08.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:41:08.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:41:08.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:41:08.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.940+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:08.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.940+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:08.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:08.940+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:13.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:13.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:13.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a240 secure :-1 s=READY pgs=13608 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc00adc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:13.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a240 secure :-1 s=READY pgs=13608 cs=0 l=1 rev1=1 rx=0x7f21dc009550 tx=0x7f21dc00adc0).stop 2021-03-09T10:41:13.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:13.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a240 unknown :-1 s=CLOSED pgs=13608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:13.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:13.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:13.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:13.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013130 con 0x7f21c400a6b0 2021-03-09T10:41:13.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc040fd0 con 0x7f21c40037a0 2021-03-09T10:41:13.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:41:13.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:13.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:13.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:13.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:13.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:13.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:13.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:13.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:13.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:13.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00af90 con 0x7f21cc0138f0 2021-03-09T10:41:13.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21cc0138f0 2021-03-09T10:41:13.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.936+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=13615 cs=0 l=1 rev1=1 rx=0x7f21cc011e90 tx=0x7f21dc005070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:13.946 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:13.940+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46787 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:13.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:41:13.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0079a0 con 0x7f21cc0138f0 2021-03-09T10:41:13.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0138f0 2021-03-09T10:41:13.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.944+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:13.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.948+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:13.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:13.948+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:18.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:18.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:18.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=13615 cs=0 l=1 rev1=1 rx=0x7f21cc011e90 tx=0x7f21dc005070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:18.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=13615 cs=0 l=1 rev1=1 rx=0x7f21cc011e90 tx=0x7f21dc005070).stop 2021-03-09T10:41:18.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:18.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=13615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:18.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:18.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:18.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:18.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21c400a6b0 2021-03-09T10:41:18.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08cf70 con 0x7f21c40037a0 2021-03-09T10:41:18.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:41:18.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:18.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:18.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:18.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:18.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.944+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:18.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:18.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:18.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:18.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:18.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc029d70 con 0x7f21c4004620 2021-03-09T10:41:18.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac60 con 0x7f21c4004620 2021-03-09T10:41:18.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.948+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a1a0 secure :-1 s=READY pgs=13630 cs=0 l=1 rev1=1 rx=0x7f21e404fb70 tx=0x7f21e407d1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:18.955 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:18.952+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46827 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:18.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40093b0 con 0x7f21c4004620 2021-03-09T10:41:18.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40093b0 con 0x7f21c4004620 2021-03-09T10:41:18.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4004620 2021-03-09T10:41:18.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.956+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:18.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.956+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:18.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:18.956+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:23.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:23.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:23.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a1a0 secure :-1 s=READY pgs=13630 cs=0 l=1 rev1=1 rx=0x7f21e404fb70 tx=0x7f21e407d1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:23.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a1a0 secure :-1 s=READY pgs=13630 cs=0 l=1 rev1=1 rx=0x7f21e404fb70 tx=0x7f21e407d1d0).stop 2021-03-09T10:41:23.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:23.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a1a0 unknown :-1 s=CLOSED pgs=13630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:23.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:23.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:23.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:23.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21c400a6b0 2021-03-09T10:41:23.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:41:23.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc008120 con 0x7f21cc0138f0 2021-03-09T10:41:23.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:23.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:23.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:23.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:23.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:23.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:23.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:23.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:23.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:23.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:41:23.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00af90 con 0x7f21cc0138f0 2021-03-09T10:41:23.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.955+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 secure :-1 s=READY pgs=13635 cs=0 l=1 rev1=1 rx=0x7f21cc053710 tx=0x7f21dc004f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:23.966 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:23.959+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46863 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:23.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.959+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007c00 con 0x7f21cc0138f0 2021-03-09T10:41:23.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.959+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:41:23.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:41:23.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.963+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:23.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.963+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:23.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:23.963+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:28.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:28.112 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:28.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:28.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:28.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 secure :-1 s=READY pgs=13635 cs=0 l=1 rev1=1 rx=0x7f21cc053710 tx=0x7f21dc004f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:28.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 secure :-1 s=READY pgs=13635 cs=0 l=1 rev1=1 rx=0x7f21cc053710 tx=0x7f21dc004f80).stop 2021-03-09T10:41:28.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:28.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 unknown :-1 s=CLOSED pgs=13635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:28.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:28.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:28.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:28.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21c400a6b0 2021-03-09T10:41:28.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:41:28.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013430 con 0x7f21c4004620 2021-03-09T10:41:28.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:28.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:28.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:28.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:28.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:28.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:28.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:28.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:28.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:28.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ae00 con 0x7f21c4004620 2021-03-09T10:41:28.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.963+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:41:28.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.967+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009fb0 secure :-1 s=READY pgs=13648 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc018200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:28.978 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:28.971+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46903 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:28.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.971+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:41:28.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.971+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21c4004620 2021-03-09T10:41:28.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:41:28.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.975+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:28.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:28.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:28.975+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:33.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:33.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:33.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009fb0 secure :-1 s=READY pgs=13648 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc018200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:33.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009fb0 secure :-1 s=READY pgs=13648 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc018200).stop 2021-03-09T10:41:33.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:33.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009fb0 unknown :-1 s=CLOSED pgs=13648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:33.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:33.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:33.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:33.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:41:33.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0216e0 con 0x7f21c40037a0 2021-03-09T10:41:33.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21cc0138f0 2021-03-09T10:41:33.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:33.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:33.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:33.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:33.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:33.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:33.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:33.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:33.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:33.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00af90 con 0x7f21cc0138f0 2021-03-09T10:41:33.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012ee0 con 0x7f21cc0138f0 2021-03-09T10:41:33.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.975+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a4a0 secure :-1 s=READY pgs=13656 cs=0 l=1 rev1=1 rx=0x7f21e00451e0 tx=0x7f21e007d5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:33.985 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:33.983+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46939 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:33.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0138f0 2021-03-09T10:41:33.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:41:33.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.987+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:41:33.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.987+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:33.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.987+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:33.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:33.987+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:38.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:38.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:38.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a4a0 secure :-1 s=READY pgs=13656 cs=0 l=1 rev1=1 rx=0x7f21e00451e0 tx=0x7f21e007d5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:38.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a4a0 secure :-1 s=READY pgs=13656 cs=0 l=1 rev1=1 rx=0x7f21e00451e0 tx=0x7f21e007d5f0).stop 2021-03-09T10:41:38.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:38.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a4a0 unknown :-1 s=CLOSED pgs=13656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:38.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012d60 con 0x7f21c400a6b0 2021-03-09T10:41:38.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21c40037a0 2021-03-09T10:41:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:41:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:38.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:38.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:38.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:38.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.983+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:38.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:38.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:38.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:38.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:41:38.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ae00 con 0x7f21c4004620 2021-03-09T10:41:38.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.987+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009dc0 secure :-1 s=READY pgs=13667 cs=0 l=1 rev1=1 rx=0x7f21cc0283d0 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:38.994 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:38.991+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46978 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:38.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005210 con 0x7f21c4004620 2021-03-09T10:41:38.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.995+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005210 con 0x7f21c4004620 2021-03-09T10:41:38.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.999+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc00bdb0 con 0x7f21c4004620 2021-03-09T10:41:38.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.999+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:39.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:39.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:38.999+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:43.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:43.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:43.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009dc0 secure :-1 s=READY pgs=13667 cs=0 l=1 rev1=1 rx=0x7f21cc0283d0 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:43.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009dc0 secure :-1 s=READY pgs=13667 cs=0 l=1 rev1=1 rx=0x7f21cc0283d0 tx=0x7f21dc010d20).stop 2021-03-09T10:41:43.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:43.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009dc0 unknown :-1 s=CLOSED pgs=13667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:43.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:43.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:43.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:43.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012d60 con 0x7f21c400a6b0 2021-03-09T10:41:43.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21c40037a0 2021-03-09T10:41:43.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.991+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc0138f0 2021-03-09T10:41:44.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:44.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:44.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:44.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:44.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:44.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:44.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:44.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:44.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:44.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc013400 con 0x7f21cc0138f0 2021-03-09T10:41:44.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.995+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:41:44.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:43.999+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=13682 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc089a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:44.007 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:44.003+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47015 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:44.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:44.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc0138f0 2021-03-09T10:41:44.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:44.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc0138f0 2021-03-09T10:41:44.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:44.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:41:44.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:44.011+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:44.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:44.011+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:44.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:44.011+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:49.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:49.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:49.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=13682 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc089a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:49.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 secure :-1 s=READY pgs=13682 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc089a30).stop 2021-03-09T10:41:49.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:49.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=13682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:49.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:49.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:49.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:49.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012d60 con 0x7f21c400a6b0 2021-03-09T10:41:49.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e490 con 0x7f21c40037a0 2021-03-09T10:41:49.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017e50 con 0x7f21c4004620 2021-03-09T10:41:49.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:49.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:49.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:49.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:49.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:49.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:49.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.003+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:49.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.007+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:49.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.007+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:49.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.007+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:41:49.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.007+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00af90 con 0x7f21c4004620 2021-03-09T10:41:49.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.007+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009bd0 secure :-1 s=READY pgs=13689 cs=0 l=1 rev1=1 rx=0x7f21e0075cf0 tx=0x7f21e000ae50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:49.017 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:49.011+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47053 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:49.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:41:49.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.015+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:41:49.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.019+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:41:49.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.019+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:49.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.019+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:49.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:49.019+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:54.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:54.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:54.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009bd0 secure :-1 s=READY pgs=13689 cs=0 l=1 rev1=1 rx=0x7f21e0075cf0 tx=0x7f21e000ae50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:54.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009bd0 secure :-1 s=READY pgs=13689 cs=0 l=1 rev1=1 rx=0x7f21e0075cf0 tx=0x7f21e000ae50).stop 2021-03-09T10:41:54.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:54.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009bd0 unknown :-1 s=CLOSED pgs=13689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:54.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:54.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:54.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:54.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00afd0 con 0x7f21c400a6b0 2021-03-09T10:41:54.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e490 con 0x7f21c40037a0 2021-03-09T10:41:54.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012f00 con 0x7f21cc0138f0 2021-03-09T10:41:54.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:54.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:54.018 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:54.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:54.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.011+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:54.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:54.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:54.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:54.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:54.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:41:54.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:41:54.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.015+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 secure :-1 s=READY pgs=13701 cs=0 l=1 rev1=1 rx=0x7f21e001f3e0 tx=0x7f21e0013960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:54.021 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:54.019+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47091 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:54.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.023+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006b30 con 0x7f21cc0138f0 2021-03-09T10:41:54.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.027+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006b30 con 0x7f21cc0138f0 2021-03-09T10:41:54.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e000a000 con 0x7f21cc0138f0 2021-03-09T10:41:54.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.031+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:54.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.031+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:54.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:54.031+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:41:58.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:58.152 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:41:59.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:41:59.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:59.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 secure :-1 s=READY pgs=13701 cs=0 l=1 rev1=1 rx=0x7f21e001f3e0 tx=0x7f21e0013960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:59.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 secure :-1 s=READY pgs=13701 cs=0 l=1 rev1=1 rx=0x7f21e001f3e0 tx=0x7f21e0013960).stop 2021-03-09T10:41:59.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00aae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:59.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 unknown :-1 s=CLOSED pgs=13701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:59.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:59.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:59.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:59.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00afd0 con 0x7f21c400a6b0 2021-03-09T10:41:59.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f070 con 0x7f21c40037a0 2021-03-09T10:41:59.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e490 con 0x7f21c4004620 2021-03-09T10:41:59.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:41:59.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:41:59.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:59.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.019+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:59.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:59.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:59.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:59.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:59.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:59.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:41:59.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:41:59.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.023+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009db0 secure :-1 s=READY pgs=13712 cs=0 l=1 rev1=1 rx=0x7f21e0013ab0 tx=0x7f21e0007de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:59.030 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:41:59.027+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47129 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:41:59.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003c230 con 0x7f21c4004620 2021-03-09T10:41:59.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21c4004620 2021-03-09T10:41:59.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e000a000 con 0x7f21c4004620 2021-03-09T10:41:59.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.035+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:41:59.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.035+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:41:59.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:41:59.035+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:04.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:42:04.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:04.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009db0 secure :-1 s=READY pgs=13712 cs=0 l=1 rev1=1 rx=0x7f21e0013ab0 tx=0x7f21e0007de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:04.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009db0 secure :-1 s=READY pgs=13712 cs=0 l=1 rev1=1 rx=0x7f21e0013ab0 tx=0x7f21e0007de0).stop 2021-03-09T10:42:04.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:04.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009db0 unknown :-1 s=CLOSED pgs=13712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:04.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:04.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.027+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:04.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:04.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f210 con 0x7f21c400a6b0 2021-03-09T10:42:04.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00afd0 con 0x7f21c40037a0 2021-03-09T10:42:04.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e490 con 0x7f21cc0138f0 2021-03-09T10:42:04.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:04.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:04.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:04.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:04.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:04.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:04.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:04.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:04.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:04.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:42:04.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:42:04.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.035+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 secure :-1 s=READY pgs=13722 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0006b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:04.039 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:04.039+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:04.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21cc0138f0 2021-03-09T10:42:04.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.043+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004e10 con 0x7f21cc0138f0 2021-03-09T10:42:04.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00263c0 con 0x7f21cc0138f0 2021-03-09T10:42:04.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.047+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:04.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.047+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:04.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:04.047+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:09.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:09.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:09.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 secure :-1 s=READY pgs=13722 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0006b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:09.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 secure :-1 s=READY pgs=13722 cs=0 l=1 rev1=1 rx=0x7f21e0005840 tx=0x7f21e0006b30).stop 2021-03-09T10:42:09.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00aae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:09.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00aae0 unknown :-1 s=CLOSED pgs=13722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:09.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:09.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:09.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:09.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095830 con 0x7f21c400a6b0 2021-03-09T10:42:09.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00afd0 con 0x7f21c40037a0 2021-03-09T10:42:09.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e490 con 0x7f21c4004620 2021-03-09T10:42:09.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:09.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:09.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:09.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:09.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.039+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:09.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:09.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:09.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:09.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:09.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08cf70 con 0x7f21c4004620 2021-03-09T10:42:09.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:42:09.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.043+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009a00 secure :-1 s=READY pgs=13736 cs=0 l=1 rev1=1 rx=0x7f21e00ae520 tx=0x7f21e0016f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:09.048 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:09.047+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:09.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00ae180 con 0x7f21c4004620 2021-03-09T10:42:09.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21c4004620 2021-03-09T10:42:09.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003f240 con 0x7f21c4004620 2021-03-09T10:42:09.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.051+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:09.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:09.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:09.055+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:14.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:14.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:14.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009a00 secure :-1 s=READY pgs=13736 cs=0 l=1 rev1=1 rx=0x7f21e00ae520 tx=0x7f21e0016f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:14.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009a00 secure :-1 s=READY pgs=13736 cs=0 l=1 rev1=1 rx=0x7f21e00ae520 tx=0x7f21e0016f20).stop 2021-03-09T10:42:14.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc009a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:14.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc009a00 unknown :-1 s=CLOSED pgs=13736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:14.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:14.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:14.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:14.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:14.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:14.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e490 con 0x7f21c400a6b0 2021-03-09T10:42:14.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0538b0 con 0x7f21c40037a0 2021-03-09T10:42:14.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095830 con 0x7f21cc0138f0 2021-03-09T10:42:14.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:14.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.047+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:14.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:14.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:14.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:14.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:14.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:14.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012f00 con 0x7f21cc0138f0 2021-03-09T10:42:14.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.051+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc013400 con 0x7f21cc0138f0 2021-03-09T10:42:14.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13743 cs=0 l=1 rev1=1 rx=0x7f21dc0053d0 tx=0x7f21dc06bf80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:14.058 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:14.055+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47243 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:14.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.059+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21cc0138f0 2021-03-09T10:42:14.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc0138f0 2021-03-09T10:42:14.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:42:14.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.063+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:14.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.067+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:14.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:14.067+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:19.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:19.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:19.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13743 cs=0 l=1 rev1=1 rx=0x7f21dc0053d0 tx=0x7f21dc06bf80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:19.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13743 cs=0 l=1 rev1=1 rx=0x7f21dc0053d0 tx=0x7f21dc06bf80).stop 2021-03-09T10:42:19.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:19.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=CLOSED pgs=13743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:19.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:19.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:19.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0538b0 con 0x7f21c400a6b0 2021-03-09T10:42:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ab90 con 0x7f21c40037a0 2021-03-09T10:42:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094770 con 0x7f21c4004620 2021-03-09T10:42:19.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:19.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:19.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:19.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:19.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:19.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e440 con 0x7f21c4004620 2021-03-09T10:42:19.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.058+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012f00 con 0x7f21c4004620 2021-03-09T10:42:19.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.062+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d0d0 secure :-1 s=READY pgs=13754 cs=0 l=1 rev1=1 rx=0x7f21cc043790 tx=0x7f21dc071f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:19.067 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:19.066+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47280 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:19.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4004620 2021-03-09T10:42:19.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4004620 2021-03-09T10:42:19.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c4004620 2021-03-09T10:42:19.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.066+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:19.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:19.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:19.070+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:24.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d0d0 secure :-1 s=READY pgs=13754 cs=0 l=1 rev1=1 rx=0x7f21cc043790 tx=0x7f21dc071f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d0d0 secure :-1 s=READY pgs=13754 cs=0 l=1 rev1=1 rx=0x7f21cc043790 tx=0x7f21dc071f10).stop 2021-03-09T10:42:24.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d0d0 unknown :-1 s=CLOSED pgs=13754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:24.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:24.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0538b0 con 0x7f21c400a6b0 2021-03-09T10:42:24.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094210 con 0x7f21c40037a0 2021-03-09T10:42:24.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:42:24.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:24.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:24.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:24.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:24.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:24.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:24.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:24.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:24.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:24.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012b20 con 0x7f21cc0138f0 2021-03-09T10:42:24.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.066+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08cf70 con 0x7f21cc0138f0 2021-03-09T10:42:24.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.070+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13763 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:24.078 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:24.074+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47319 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:24.078 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a4510 con 0x7f21cc0138f0 2021-03-09T10:42:24.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.074+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:42:24.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:42:24.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.078+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:24.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.078+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:24.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:24.078+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:28.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:28.200 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:29.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:29.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:29.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13763 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:29.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13763 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e000b450).stop 2021-03-09T10:42:29.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:29.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=CLOSED pgs=13763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:29.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:29.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:29.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:29.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d520 con 0x7f21c400a6b0 2021-03-09T10:42:29.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0538b0 con 0x7f21c40037a0 2021-03-09T10:42:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:42:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:29.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:29.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:29.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:29.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:29.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:29.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00aac0 con 0x7f21c4004620 2021-03-09T10:42:29.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.078+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e440 con 0x7f21c4004620 2021-03-09T10:42:29.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.082+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13776 cs=0 l=1 rev1=1 rx=0x7f21dc005250 tx=0x7f21dc0098a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:29.207 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:29.086+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47356 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:29.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0193e0 con 0x7f21c4004620 2021-03-09T10:42:29.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21c4004620 2021-03-09T10:42:29.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4004620 2021-03-09T10:42:29.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.094+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:29.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.094+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:29.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:29.094+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:34.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:42:34.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:34.089 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13776 cs=0 l=1 rev1=1 rx=0x7f21dc005250 tx=0x7f21dc0098a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:34.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13776 cs=0 l=1 rev1=1 rx=0x7f21dc005250 tx=0x7f21dc0098a0).stop 2021-03-09T10:42:34.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:34.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=CLOSED pgs=13776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:34.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:34.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:34.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:34.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0538b0 con 0x7f21c400a6b0 2021-03-09T10:42:34.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:42:34.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:42:34.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:34.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:34.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:34.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:34.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:34.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:34.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac50 con 0x7f21cc0138f0 2021-03-09T10:42:34.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.086+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012b20 con 0x7f21cc0138f0 2021-03-09T10:42:34.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.090+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13784 cs=0 l=1 rev1=1 rx=0x7f21e0013a60 tx=0x7f21e00136c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:34.097 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:34.094+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47395 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:34.097 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0037e90 con 0x7f21cc0138f0 2021-03-09T10:42:34.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0037e90 con 0x7f21cc0138f0 2021-03-09T10:42:34.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.098+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:42:34.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.098+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:34.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.098+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:34.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:34.098+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:39.098 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:39.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:39.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13784 cs=0 l=1 rev1=1 rx=0x7f21e0013a60 tx=0x7f21e00136c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:39.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 secure :-1 s=READY pgs=13784 cs=0 l=1 rev1=1 rx=0x7f21e0013a60 tx=0x7f21e00136c0).stop 2021-03-09T10:42:39.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc009ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:39.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc009ef0 unknown :-1 s=CLOSED pgs=13784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:39.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:39.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:39.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:39.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:42:39.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012980 con 0x7f21c40037a0 2021-03-09T10:42:39.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ac0 con 0x7f21c4004620 2021-03-09T10:42:39.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:39.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:39.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:39.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:39.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.094+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:39.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:39.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:39.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:39.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:39.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc012c00 con 0x7f21c4004620 2021-03-09T10:42:39.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084030 con 0x7f21c4004620 2021-03-09T10:42:39.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.098+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13795 cs=0 l=1 rev1=1 rx=0x7f21e0013a10 tx=0x7f21e00a4190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:39.106 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:39.102+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:39.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.106+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0084030 con 0x7f21c4004620 2021-03-09T10:42:39.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0084030 con 0x7f21c4004620 2021-03-09T10:42:39.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.110+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:42:39.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.110+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:39.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.114+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:39.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:39.114+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:44.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:44.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:44.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13795 cs=0 l=1 rev1=1 rx=0x7f21e0013a10 tx=0x7f21e00a4190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:44.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13795 cs=0 l=1 rev1=1 rx=0x7f21e0013a10 tx=0x7f21e00a4190).stop 2021-03-09T10:42:44.108 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:44.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=CLOSED pgs=13795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:44.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.102+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:44.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:44.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:44.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:42:44.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a280 con 0x7f21c40037a0 2021-03-09T10:42:44.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021da0 con 0x7f21cc0138f0 2021-03-09T10:42:44.110 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:44.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:44.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:44.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:44.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:44.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:44.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:44.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:44.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:44.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:42:44.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00aac0 con 0x7f21cc0138f0 2021-03-09T10:42:44.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.106+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 secure :-1 s=READY pgs=13806 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc072280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:44.117 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:44.114+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47471 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:44.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0138f0 2021-03-09T10:42:44.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc0138f0 2021-03-09T10:42:44.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.118+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21cc0138f0 2021-03-09T10:42:44.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.118+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:44.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.118+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:44.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:44.118+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:49.118 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:42:49.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:49.119 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 secure :-1 s=READY pgs=13806 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc072280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:49.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 secure :-1 s=READY pgs=13806 cs=0 l=1 rev1=1 rx=0x7f21dc015de0 tx=0x7f21dc072280).stop 2021-03-09T10:42:49.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00ee80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:49.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 unknown :-1 s=CLOSED pgs=13806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:49.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:49.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:49.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:49.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0970a0 con 0x7f21c400a6b0 2021-03-09T10:42:49.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:42:49.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f4e0 con 0x7f21c4004620 2021-03-09T10:42:49.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:49.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:49.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:49.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:49.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.114+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:49.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:49.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:49.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:49.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:49.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f370 con 0x7f21c4004620 2021-03-09T10:42:49.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:42:49.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.118+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13817 cs=0 l=1 rev1=1 rx=0x7f21dc063050 tx=0x7f21dc019220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:49.130 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:49.122+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47504 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:49.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083220 con 0x7f21c4004620 2021-03-09T10:42:49.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.122+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083220 con 0x7f21c4004620 2021-03-09T10:42:49.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.130+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c4004620 2021-03-09T10:42:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.130+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.130+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:49.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:49.130+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:54.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:54.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:54.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13817 cs=0 l=1 rev1=1 rx=0x7f21dc063050 tx=0x7f21dc019220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:54.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13817 cs=0 l=1 rev1=1 rx=0x7f21dc063050 tx=0x7f21dc019220).stop 2021-03-09T10:42:54.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:54.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=CLOSED pgs=13817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:54.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:54.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:54.128 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:54.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042dd0 con 0x7f21c400a6b0 2021-03-09T10:42:54.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:42:54.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0970a0 con 0x7f21cc0138f0 2021-03-09T10:42:54.129 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:54.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:54.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:54.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:54.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:54.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:54.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:54.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:54.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:54.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:42:54.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc012c00 con 0x7f21cc0138f0 2021-03-09T10:42:54.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.126+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 secure :-1 s=READY pgs=13828 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e002b250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:54.134 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:54.130+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47542 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:54.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:42:54.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:42:54.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.138+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:42:54.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.138+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:54.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.138+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:54.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:54.138+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:42:58.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:58.247 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:42:59.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:59.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:59.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 secure :-1 s=READY pgs=13828 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e002b250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:59.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 secure :-1 s=READY pgs=13828 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e002b250).stop 2021-03-09T10:42:59.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00ee80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:59.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00ee80 unknown :-1 s=CLOSED pgs=13828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:59.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:59.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:59.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:59.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:42:59.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae20 con 0x7f21c40037a0 2021-03-09T10:42:59.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:42:59.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f6a0 con 0x7f21c4004620 2021-03-09T10:42:59.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:59.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:42:59.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:59.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:59.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:59.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc097050 con 0x7f21c4004620 2021-03-09T10:42:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.134+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f370 con 0x7f21c4004620 2021-03-09T10:42:59.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.138+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13839 cs=0 l=1 rev1=1 rx=0x7f21cc01ad20 tx=0x7f21dc06bec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:59.143 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:42:59.142+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47579 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:42:59.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21c4004620 2021-03-09T10:42:59.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21c4004620 2021-03-09T10:42:59.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4004620 2021-03-09T10:42:59.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.150+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:42:59.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.150+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:42:59.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:42:59.150+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:04.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:43:04.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:04.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13839 cs=0 l=1 rev1=1 rx=0x7f21cc01ad20 tx=0x7f21dc06bec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:04.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13839 cs=0 l=1 rev1=1 rx=0x7f21cc01ad20 tx=0x7f21dc06bec0).stop 2021-03-09T10:43:04.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:04.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=CLOSED pgs=13839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:04.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:04.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:04.147 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:43:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae20 con 0x7f21c40037a0 2021-03-09T10:43:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00edc0 con 0x7f21cc0138f0 2021-03-09T10:43:04.148 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:04.149 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:04.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:04.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:04.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:04.150 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ef00 con 0x7f21cc0138f0 2021-03-09T10:43:04.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.142+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc097050 con 0x7f21cc0138f0 2021-03-09T10:43:04.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.146+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042a40 secure :-1 s=READY pgs=13849 cs=0 l=1 rev1=1 rx=0x7f21dc063050 tx=0x7f21dc08a200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:04.152 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:04.150+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47618 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:04.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc071cc0 con 0x7f21cc0138f0 2021-03-09T10:43:04.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.150+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0138f0 2021-03-09T10:43:04.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.154+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071cc0 con 0x7f21cc0138f0 2021-03-09T10:43:04.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.154+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:04.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:04.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:04.154+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:09.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042a40 secure :-1 s=READY pgs=13849 cs=0 l=1 rev1=1 rx=0x7f21dc063050 tx=0x7f21dc08a200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042a40 secure :-1 s=READY pgs=13849 cs=0 l=1 rev1=1 rx=0x7f21dc063050 tx=0x7f21dc08a200).stop 2021-03-09T10:43:09.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:09.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042a40 unknown :-1 s=CLOSED pgs=13849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:09.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:09.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:09.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:09.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae20 con 0x7f21c400a6b0 2021-03-09T10:43:09.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:09.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:43:09.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ed80 con 0x7f21c4004620 2021-03-09T10:43:09.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:09.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:09.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:09.158 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.149+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:09.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:09.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:09.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:09.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:09.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f090 con 0x7f21c4004620 2021-03-09T10:43:09.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ef00 con 0x7f21c4004620 2021-03-09T10:43:09.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.153+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13863 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0a0c70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:09.161 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:09.157+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47655 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:09.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.157+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b31f0 con 0x7f21c4004620 2021-03-09T10:43:09.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.161+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21c4004620 2021-03-09T10:43:09.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c4004620 2021-03-09T10:43:09.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.165+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:09.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.165+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:09.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:09.165+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:14.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:43:14.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:14.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13863 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0a0c70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:14.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13863 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0a0c70).stop 2021-03-09T10:43:14.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:14.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=CLOSED pgs=13863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:14.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a220 con 0x7f21c400a6b0 2021-03-09T10:43:14.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042e90 con 0x7f21c40037a0 2021-03-09T10:43:14.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f150 con 0x7f21cc0138f0 2021-03-09T10:43:14.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:14.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.157+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:14.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:14.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:14.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:14.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:14.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:14.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:14.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:14.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc022660 con 0x7f21cc0138f0 2021-03-09T10:43:14.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f090 con 0x7f21cc0138f0 2021-03-09T10:43:14.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.161+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0429a0 secure :-1 s=READY pgs=13870 cs=0 l=1 rev1=1 rx=0x7f21dc062460 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:14.170 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:14.165+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47694 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:14.171 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.165+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc09bec0 con 0x7f21cc0138f0 2021-03-09T10:43:14.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09bec0 con 0x7f21cc0138f0 2021-03-09T10:43:14.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.173+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:43:14.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.173+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:14.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:14.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:14.173+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:19.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:43:19.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.165+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:19.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0429a0 secure :-1 s=READY pgs=13870 cs=0 l=1 rev1=1 rx=0x7f21dc062460 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:19.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.165+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0429a0 secure :-1 s=READY pgs=13870 cs=0 l=1 rev1=1 rx=0x7f21dc062460 tx=0x7f21dc009290).stop 2021-03-09T10:43:19.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0429a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:19.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0429a0 unknown :-1 s=CLOSED pgs=13870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:19.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:19.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:19.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:19.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05a650 con 0x7f21c400a6b0 2021-03-09T10:43:19.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02f150 con 0x7f21c40037a0 2021-03-09T10:43:19.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a220 con 0x7f21c4004620 2021-03-09T10:43:19.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:19.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:19.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:19.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:19.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:19.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:19.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:19.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:19.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:19.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc053710 con 0x7f21c4004620 2021-03-09T10:43:19.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:43:19.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.169+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13880 cs=0 l=1 rev1=1 rx=0x7f21e0004460 tx=0x7f21e007d540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:19.180 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:19.177+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47731 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:19.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:43:19.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.177+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:43:19.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:43:19.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.181+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:19.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:19.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:19.181+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:24.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:24.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:24.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13880 cs=0 l=1 rev1=1 rx=0x7f21e0004460 tx=0x7f21e007d540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:24.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 secure :-1 s=READY pgs=13880 cs=0 l=1 rev1=1 rx=0x7f21e0004460 tx=0x7f21e007d540).stop 2021-03-09T10:43:24.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:24.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01d030 unknown :-1 s=CLOSED pgs=13880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:24.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:24.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:24.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:24.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:24.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:24.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a220 con 0x7f21c400a6b0 2021-03-09T10:43:24.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc096030 con 0x7f21c40037a0 2021-03-09T10:43:24.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc05a620 con 0x7f21cc0138f0 2021-03-09T10:43:24.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:24.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:24.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:24.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:24.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:24.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:24.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:24.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:43:24.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc029d70 con 0x7f21cc0138f0 2021-03-09T10:43:24.188 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.181+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042930 secure :-1 s=READY pgs=13893 cs=0 l=1 rev1=1 rx=0x7f21cc057a20 tx=0x7f21e4016130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:24.190 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:24.185+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47770 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:24.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009410 con 0x7f21cc0138f0 2021-03-09T10:43:24.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009410 con 0x7f21cc0138f0 2021-03-09T10:43:24.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.193+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008ac0 con 0x7f21cc0138f0 2021-03-09T10:43:24.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.193+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:24.199 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.197+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:24.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:24.197+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:28.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:28.302 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:29.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:29.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:29.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042930 secure :-1 s=READY pgs=13893 cs=0 l=1 rev1=1 rx=0x7f21cc057a20 tx=0x7f21e4016130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:29.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042930 secure :-1 s=READY pgs=13893 cs=0 l=1 rev1=1 rx=0x7f21cc057a20 tx=0x7f21e4016130).stop 2021-03-09T10:43:29.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc042930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:29.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc042930 unknown :-1 s=CLOSED pgs=13893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:29.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:29.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:29.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:29.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053980 con 0x7f21c400a6b0 2021-03-09T10:43:29.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a080 con 0x7f21c40037a0 2021-03-09T10:43:29.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a1c0 con 0x7f21c4004620 2021-03-09T10:43:29.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:29.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:29.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:29.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:29.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:29.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:29.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:29.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:29.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:29.197 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0264c0 con 0x7f21c4004620 2021-03-09T10:43:29.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc022660 con 0x7f21c4004620 2021-03-09T10:43:29.198 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.189+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cfc0 secure :-1 s=READY pgs=13902 cs=0 l=1 rev1=1 rx=0x7f21dc033220 tx=0x7f21dc029400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:29.199 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:29.197+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:29.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc070dc0 con 0x7f21c4004620 2021-03-09T10:43:29.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc033250 con 0x7f21c4004620 2021-03-09T10:43:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.201+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:43:29.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.201+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:29.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:29.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:29.201+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:34.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:34.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:34.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cfc0 secure :-1 s=READY pgs=13902 cs=0 l=1 rev1=1 rx=0x7f21dc033220 tx=0x7f21dc029400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:34.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cfc0 secure :-1 s=READY pgs=13902 cs=0 l=1 rev1=1 rx=0x7f21dc033220 tx=0x7f21dc029400).stop 2021-03-09T10:43:34.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cfc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:34.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cfc0 unknown :-1 s=CLOSED pgs=13902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:34.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:34.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:34.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:34.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042da0 con 0x7f21c400a6b0 2021-03-09T10:43:34.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039ee0 con 0x7f21c40037a0 2021-03-09T10:43:34.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a020 con 0x7f21cc0138f0 2021-03-09T10:43:34.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:34.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:34.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:34.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:34.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.197+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:34.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:34.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:34.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:34.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:34.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03a190 con 0x7f21cc0138f0 2021-03-09T10:43:34.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:43:34.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.201+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 secure :-1 s=READY pgs=13912 cs=0 l=1 rev1=1 rx=0x7f21cc0130e0 tx=0x7f21e4021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:34.209 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:34.205+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47846 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:34.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.205+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005cf0 con 0x7f21cc0138f0 2021-03-09T10:43:34.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:43:34.218 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.213+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4005cf0 con 0x7f21cc0138f0 2021-03-09T10:43:34.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.213+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:34.219 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.217+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:34.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:34.217+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:39.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:39.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 secure :-1 s=READY pgs=13912 cs=0 l=1 rev1=1 rx=0x7f21cc0130e0 tx=0x7f21e4021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 secure :-1 s=READY pgs=13912 cs=0 l=1 rev1=1 rx=0x7f21cc0130e0 tx=0x7f21e4021000).stop 2021-03-09T10:43:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0428b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:39.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 unknown :-1 s=CLOSED pgs=13912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:39.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:39.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:39.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:39.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a160 con 0x7f21c400a6b0 2021-03-09T10:43:39.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e390 con 0x7f21c40037a0 2021-03-09T10:43:39.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e4d0 con 0x7f21c4004620 2021-03-09T10:43:39.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:39.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:39.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:39.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:39.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:39.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:39.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:39.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:39.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:39.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04e640 con 0x7f21c4004620 2021-03-09T10:43:39.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0264c0 con 0x7f21c4004620 2021-03-09T10:43:39.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.209+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 secure :-1 s=READY pgs=13923 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc033250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:39.278 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:39.217+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47883 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:39.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:43:39.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:43:39.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.221+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4004620 2021-03-09T10:43:39.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.221+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:39.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.221+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:39.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:39.221+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:44.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:43:44.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:44.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 secure :-1 s=READY pgs=13923 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc033250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:44.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 secure :-1 s=READY pgs=13923 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc033250).stop 2021-03-09T10:43:44.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:44.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 unknown :-1 s=CLOSED pgs=13923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:44.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:44.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:44.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092730 con 0x7f21c400a6b0 2021-03-09T10:43:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042da0 con 0x7f21c40037a0 2021-03-09T10:43:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039cf0 con 0x7f21cc0138f0 2021-03-09T10:43:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:44.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:44.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:44.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:44.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:44.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039e60 con 0x7f21cc0138f0 2021-03-09T10:43:44.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.217+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:43:44.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.221+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 secure :-1 s=READY pgs=13934 cs=0 l=1 rev1=1 rx=0x7f21cc041480 tx=0x7f21e404ff80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:44.229 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:44.225+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47922 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:44.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40417a0 con 0x7f21cc0138f0 2021-03-09T10:43:44.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b0120 con 0x7f21cc0138f0 2021-03-09T10:43:44.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.233+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:43:44.238 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.233+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:44.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.237+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:44.239 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:44.237+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:49.230 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:49.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:49.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 secure :-1 s=READY pgs=13934 cs=0 l=1 rev1=1 rx=0x7f21cc041480 tx=0x7f21e404ff80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:49.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 secure :-1 s=READY pgs=13934 cs=0 l=1 rev1=1 rx=0x7f21cc041480 tx=0x7f21e404ff80).stop 2021-03-09T10:43:49.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0428b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:49.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0428b0 unknown :-1 s=CLOSED pgs=13934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:49.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:49.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:49.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:49.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092730 con 0x7f21c400a6b0 2021-03-09T10:43:49.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039cf0 con 0x7f21c40037a0 2021-03-09T10:43:49.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b2c0 con 0x7f21c4004620 2021-03-09T10:43:49.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:49.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:49.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:49.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:49.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:49.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:49.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:49.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:49.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:49.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f060 con 0x7f21c4004620 2021-03-09T10:43:49.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.229+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc053710 con 0x7f21c4004620 2021-03-09T10:43:49.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.233+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 secure :-1 s=READY pgs=13949 cs=0 l=1 rev1=1 rx=0x7f21e009aab0 tx=0x7f21e0029110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:49.240 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:49.237+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47959 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:49.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:43:49.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.241+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:43:49.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:43:49.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.245+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:49.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.245+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:49.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:49.245+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:54.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:54.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:54.241 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 secure :-1 s=READY pgs=13949 cs=0 l=1 rev1=1 rx=0x7f21e009aab0 tx=0x7f21e0029110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:54.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 secure :-1 s=READY pgs=13949 cs=0 l=1 rev1=1 rx=0x7f21e009aab0 tx=0x7f21e0029110).stop 2021-03-09T10:43:54.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:54.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf70 unknown :-1 s=CLOSED pgs=13949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:54.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:54.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:54.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:54.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023620 con 0x7f21c400a6b0 2021-03-09T10:43:54.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039cf0 con 0x7f21c40037a0 2021-03-09T10:43:54.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094860 con 0x7f21cc0138f0 2021-03-09T10:43:54.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:54.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:54.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:54.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:54.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.237+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:54.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:54.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:54.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:54.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:54.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092730 con 0x7f21cc0138f0 2021-03-09T10:43:54.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039e60 con 0x7f21cc0138f0 2021-03-09T10:43:54.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.241+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=13955 cs=0 l=1 rev1=1 rx=0x7f21e40b02e0 tx=0x7f21e4036170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:54.249 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:54.245+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47998 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:54.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407d0f0 con 0x7f21cc0138f0 2021-03-09T10:43:54.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.245+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4053ed0 con 0x7f21cc0138f0 2021-03-09T10:43:54.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.249+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4041220 con 0x7f21cc0138f0 2021-03-09T10:43:54.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.249+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:54.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.249+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:54.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:54.249+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:43:58.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:58.375 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:43:59.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:59.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:59.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=13955 cs=0 l=1 rev1=1 rx=0x7f21e40b02e0 tx=0x7f21e4036170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:59.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=13955 cs=0 l=1 rev1=1 rx=0x7f21e40b02e0 tx=0x7f21e4036170).stop 2021-03-09T10:43:59.251 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:59.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=CLOSED pgs=13955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:59.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:59.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:59.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:59.252 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c400a6b0 2021-03-09T10:43:59.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c810 con 0x7f21c40037a0 2021-03-09T10:43:59.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039cf0 con 0x7f21c4004620 2021-03-09T10:43:59.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:43:59.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:59.253 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:43:59.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:59.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:59.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:59.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:59.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:59.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:59.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0462c0 con 0x7f21c4004620 2021-03-09T10:43:59.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f060 con 0x7f21c4004620 2021-03-09T10:43:59.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.248+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=13966 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0090080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:59.258 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:43:59.252+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48035 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:43:59.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.252+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0005f20 con 0x7f21c4004620 2021-03-09T10:43:59.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:43:59.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.260+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:43:59.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.260+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:43:59.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.260+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:43:59.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:43:59.260+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:04.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:04.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:04.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=13966 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0090080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:04.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=13966 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e0090080).stop 2021-03-09T10:44:04.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:04.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=CLOSED pgs=13966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:04.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:04.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:04.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:04.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c400a6b0 2021-03-09T10:44:04.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094860 con 0x7f21c40037a0 2021-03-09T10:44:04.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049b20 con 0x7f21cc0138f0 2021-03-09T10:44:04.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:04.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:04.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:04.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:04.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:04.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:04.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:04.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:04.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:04.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:44:04.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04e640 con 0x7f21cc0138f0 2021-03-09T10:44:04.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.256+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04af30 secure :-1 s=READY pgs=13976 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:04.271 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:04.264+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48073 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:04.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068840 con 0x7f21cc0138f0 2021-03-09T10:44:04.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.264+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21cc0138f0 2021-03-09T10:44:04.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029120 con 0x7f21cc0138f0 2021-03-09T10:44:04.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.268+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:04.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.268+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:04.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:04.268+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:09.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:44:09.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:09.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04af30 secure :-1 s=READY pgs=13976 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:09.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04af30 secure :-1 s=READY pgs=13976 cs=0 l=1 rev1=1 rx=0x7f21dc009700 tx=0x7f21dc010d90).stop 2021-03-09T10:44:09.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04af30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:09.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04af30 unknown :-1 s=CLOSED pgs=13976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:09.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:09.271 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:09.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:09.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c400a6b0 2021-03-09T10:44:09.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094860 con 0x7f21c40037a0 2021-03-09T10:44:09.272 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049980 con 0x7f21c4004620 2021-03-09T10:44:09.273 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:09.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:09.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:09.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:09.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:09.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:09.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:09.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:09.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:09.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc049ac0 con 0x7f21c4004620 2021-03-09T10:44:09.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:44:09.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.268+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=13987 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc0811c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:09.277 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:09.272+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48110 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:09.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.272+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0330f0 con 0x7f21c4004620 2021-03-09T10:44:09.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089dc0 con 0x7f21c4004620 2021-03-09T10:44:09.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21c4004620 2021-03-09T10:44:09.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.276+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:09.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.276+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:09.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:09.276+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:14.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:44:14.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:14.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=13987 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc0811c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:14.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=13987 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc0811c0).stop 2021-03-09T10:44:14.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:14.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=CLOSED pgs=13987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:14.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:14.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:14.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:14.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c400a6b0 2021-03-09T10:44:14.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025f00 con 0x7f21c40037a0 2021-03-09T10:44:14.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049930 con 0x7f21cc0138f0 2021-03-09T10:44:14.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:14.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:14.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:14.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:14.283 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:14.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:14.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:14.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:14.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:14.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:44:14.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.276+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0462c0 con 0x7f21cc0138f0 2021-03-09T10:44:14.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.280+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04aeb0 secure :-1 s=READY pgs=13999 cs=0 l=1 rev1=1 rx=0x7f21e00a5d80 tx=0x7f21e000b450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:14.286 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:14.280+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48148 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:14.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:44:14.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:44:14.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.288+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:44:14.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.288+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:14.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:14.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:14.288+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:19.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:19.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:19.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04aeb0 secure :-1 s=READY pgs=13999 cs=0 l=1 rev1=1 rx=0x7f21e00a5d80 tx=0x7f21e000b450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:19.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04aeb0 secure :-1 s=READY pgs=13999 cs=0 l=1 rev1=1 rx=0x7f21e00a5d80 tx=0x7f21e000b450).stop 2021-03-09T10:44:19.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04aeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:19.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04aeb0 unknown :-1 s=CLOSED pgs=13999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:19.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:19.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:19.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:19.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c400a6b0 2021-03-09T10:44:19.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046250 con 0x7f21c40037a0 2021-03-09T10:44:19.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046390 con 0x7f21c4004620 2021-03-09T10:44:19.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:19.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:19.292 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:19.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:19.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:19.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:19.293 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:19.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:19.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:19.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.284+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039cf0 con 0x7f21c4004620 2021-03-09T10:44:19.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.288+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc049ac0 con 0x7f21c4004620 2021-03-09T10:44:19.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.288+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14010 cs=0 l=1 rev1=1 rx=0x7f21cc00a7f0 tx=0x7f21dc0330f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:19.296 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:19.292+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48186 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:19.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069660 con 0x7f21c4004620 2021-03-09T10:44:19.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.292+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069660 con 0x7f21c4004620 2021-03-09T10:44:19.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4004620 2021-03-09T10:44:19.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.296+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:19.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.296+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:19.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:19.296+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:24.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:24.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:24.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14010 cs=0 l=1 rev1=1 rx=0x7f21cc00a7f0 tx=0x7f21dc0330f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:24.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14010 cs=0 l=1 rev1=1 rx=0x7f21cc00a7f0 tx=0x7f21dc0330f0).stop 2021-03-09T10:44:24.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:24.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=CLOSED pgs=14010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:24.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:24.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:24.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:24.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c400a6b0 2021-03-09T10:44:24.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a850 con 0x7f21c40037a0 2021-03-09T10:44:24.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a990 con 0x7f21cc0138f0 2021-03-09T10:44:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:24.304 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:24.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:24.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:24.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:24.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:24.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:24.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:24.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04aeb0 con 0x7f21cc0138f0 2021-03-09T10:44:24.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc092730 con 0x7f21cc0138f0 2021-03-09T10:44:24.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.296+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 secure :-1 s=READY pgs=14019 cs=0 l=1 rev1=1 rx=0x7f21e40093c0 tx=0x7f21e4009460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:24.312 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:24.300+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48224 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:24.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.300+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a65d0 con 0x7f21cc0138f0 2021-03-09T10:44:24.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a65d0 con 0x7f21cc0138f0 2021-03-09T10:44:24.314 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.308+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:44:24.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.308+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:24.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:24.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:24.308+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:28.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:28.414 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc046250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc046250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:29.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 secure :-1 s=READY pgs=14019 cs=0 l=1 rev1=1 rx=0x7f21e40093c0 tx=0x7f21e4009460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 secure :-1 s=READY pgs=14019 cs=0 l=1 rev1=1 rx=0x7f21e40093c0 tx=0x7f21e4009460).stop 2021-03-09T10:44:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc046250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 unknown :-1 s=CLOSED pgs=14019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:29.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:29.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:29.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092770 con 0x7f21c400a6b0 2021-03-09T10:44:29.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056df0 con 0x7f21c40037a0 2021-03-09T10:44:29.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c810 con 0x7f21c4004620 2021-03-09T10:44:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:29.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:29.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:29.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04c950 con 0x7f21c4004620 2021-03-09T10:44:29.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:44:29.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.304+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14030 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003c0b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:29.314 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:29.308+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48262 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:29.315 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.312+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003c230 con 0x7f21c4004620 2021-03-09T10:44:29.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.316+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003c230 con 0x7f21c4004620 2021-03-09T10:44:29.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.316+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003dd30 con 0x7f21c4004620 2021-03-09T10:44:29.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.316+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:29.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.316+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:29.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:29.316+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14030 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003c0b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:34.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14030 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003c0b0).stop 2021-03-09T10:44:34.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:34.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=CLOSED pgs=14030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:34.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:34.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:34.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:34.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a190 con 0x7f21c400a6b0 2021-03-09T10:44:34.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092730 con 0x7f21c40037a0 2021-03-09T10:44:34.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:44:34.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:34.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:34.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:34.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:34.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:34.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:34.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:34.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:34.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:34.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc047690 con 0x7f21cc0138f0 2021-03-09T10:44:34.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.312+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc0138f0 2021-03-09T10:44:34.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.316+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 secure :-1 s=READY pgs=14040 cs=0 l=1 rev1=1 rx=0x7f21e000b420 tx=0x7f21e0015ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:34.323 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:34.320+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:34.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.320+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e002a100 con 0x7f21cc0138f0 2021-03-09T10:44:34.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.324+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00aed40 con 0x7f21cc0138f0 2021-03-09T10:44:34.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.324+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:44:34.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.328+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:34.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.328+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:34.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:34.328+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:39.325 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc046250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc046250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 secure :-1 s=READY pgs=14040 cs=0 l=1 rev1=1 rx=0x7f21e000b420 tx=0x7f21e0015ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 secure :-1 s=READY pgs=14040 cs=0 l=1 rev1=1 rx=0x7f21e000b420 tx=0x7f21e0015ee0).stop 2021-03-09T10:44:39.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc046250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:39.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc046250 unknown :-1 s=CLOSED pgs=14040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:39.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:39.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:39.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:39.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a190 con 0x7f21c400a6b0 2021-03-09T10:44:39.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013110 con 0x7f21c40037a0 2021-03-09T10:44:39.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b040 con 0x7f21c4004620 2021-03-09T10:44:39.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:39.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:39.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:39.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:39.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:39.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:39.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:39.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:39.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:39.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04b180 con 0x7f21c4004620 2021-03-09T10:44:39.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04aeb0 con 0x7f21c4004620 2021-03-09T10:44:39.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.324+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14051 cs=0 l=1 rev1=1 rx=0x7f21cc0221f0 tx=0x7f21e40039a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:39.333 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:39.328+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48338 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:39.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.332+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:44:39.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.336+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40282e0 con 0x7f21c4004620 2021-03-09T10:44:39.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.336+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:44:39.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.336+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:39.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.336+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:39.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:39.336+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:44.334 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:44.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:44.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14051 cs=0 l=1 rev1=1 rx=0x7f21cc0221f0 tx=0x7f21e40039a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:44.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14051 cs=0 l=1 rev1=1 rx=0x7f21cc0221f0 tx=0x7f21e40039a0).stop 2021-03-09T10:44:44.335 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:44.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=CLOSED pgs=14051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:44.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:44.336 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:44.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:44.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0270a0 con 0x7f21c400a6b0 2021-03-09T10:44:44.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a190 con 0x7f21c40037a0 2021-03-09T10:44:44.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0138f0 2021-03-09T10:44:44.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:44.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:44.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:44.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:44.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:44.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:44.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:44.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:44.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:44.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025c10 con 0x7f21cc0138f0 2021-03-09T10:44:44.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04b180 con 0x7f21cc0138f0 2021-03-09T10:44:44.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.332+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 secure :-1 s=READY pgs=14062 cs=0 l=1 rev1=1 rx=0x7f21e4017510 tx=0x7f21e4017190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:44.343 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:44.340+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48376 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:44.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21cc0138f0 2021-03-09T10:44:44.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b12b0 con 0x7f21cc0138f0 2021-03-09T10:44:44.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008ac0 con 0x7f21cc0138f0 2021-03-09T10:44:44.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.340+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:44.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.344+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:44.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:44.344+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:49.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:49.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:49.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 secure :-1 s=READY pgs=14062 cs=0 l=1 rev1=1 rx=0x7f21e4017510 tx=0x7f21e4017190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:49.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 secure :-1 s=READY pgs=14062 cs=0 l=1 rev1=1 rx=0x7f21e4017510 tx=0x7f21e4017190).stop 2021-03-09T10:44:49.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01c180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:49.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01c180 unknown :-1 s=CLOSED pgs=14062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:49.346 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:49.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:49.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:49.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a1b0 con 0x7f21c400a6b0 2021-03-09T10:44:49.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097560 con 0x7f21c40037a0 2021-03-09T10:44:49.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0976a0 con 0x7f21c4004620 2021-03-09T10:44:49.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:49.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:49.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:49.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:49.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:49.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:49.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:49.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:49.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:49.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046250 con 0x7f21c4004620 2021-03-09T10:44:49.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.340+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039cf0 con 0x7f21c4004620 2021-03-09T10:44:49.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.344+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14072 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0725f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:49.352 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:49.348+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48414 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:49.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.348+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:44:49.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.348+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:44:49.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.348+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:44:49.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.348+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:49.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.352+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:49.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:49.352+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:54.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:54.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:54.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14072 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0725f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:54.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 secure :-1 s=READY pgs=14072 cs=0 l=1 rev1=1 rx=0x7f21dc071f50 tx=0x7f21dc0725f0).stop 2021-03-09T10:44:54.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc01cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:54.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc01cf20 unknown :-1 s=CLOSED pgs=14072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:54.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:54.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:54.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:54.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094e60 con 0x7f21c400a6b0 2021-03-09T10:44:54.355 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039e90 con 0x7f21c40037a0 2021-03-09T10:44:54.356 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0977e0 con 0x7f21cc0138f0 2021-03-09T10:44:54.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:54.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:54.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:54.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:54.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.347+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:54.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:54.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:54.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:54.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:54.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042cb0 con 0x7f21cc0138f0 2021-03-09T10:44:54.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc047690 con 0x7f21cc0138f0 2021-03-09T10:44:54.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.351+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14083 cs=0 l=1 rev1=1 rx=0x7f21e003b040 tx=0x7f21e00a4240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:54.360 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:54.355+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48452 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:54.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.355+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:44:54.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.355+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:44:54.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:44:54.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.359+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:54.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.359+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:54.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:54.359+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:44:58.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:58.462 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:44:59.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:59.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:59.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14083 cs=0 l=1 rev1=1 rx=0x7f21e003b040 tx=0x7f21e00a4240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:59.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14083 cs=0 l=1 rev1=1 rx=0x7f21e003b040 tx=0x7f21e00a4240).stop 2021-03-09T10:44:59.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:59.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=CLOSED pgs=14083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:59.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:59.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:59.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:59.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00add0 con 0x7f21c400a6b0 2021-03-09T10:44:59.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094e40 con 0x7f21c40037a0 2021-03-09T10:44:59.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039e90 con 0x7f21c4004620 2021-03-09T10:44:59.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:44:59.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:59.365 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:44:59.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:59.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:59.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:59.366 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:59.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:59.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:59.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cf20 con 0x7f21c4004620 2021-03-09T10:44:59.367 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025c10 con 0x7f21c4004620 2021-03-09T10:44:59.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.359+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0463e0 secure :-1 s=READY pgs=14094 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4048ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:59.369 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:44:59.363+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48490 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:44:59.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.363+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21c4004620 2021-03-09T10:44:59.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.371+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400fe60 con 0x7f21c4004620 2021-03-09T10:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.371+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408fbf0 con 0x7f21c4004620 2021-03-09T10:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.371+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:44:59.377 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.371+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:44:59.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:44:59.371+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:04.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:04.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:04.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0463e0 secure :-1 s=READY pgs=14094 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4048ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:04.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0463e0 secure :-1 s=READY pgs=14094 cs=0 l=1 rev1=1 rx=0x7f21e4000f40 tx=0x7f21e4048ad0).stop 2021-03-09T10:45:04.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0463e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:04.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0463e0 unknown :-1 s=CLOSED pgs=14094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:04.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:04.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:04.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:04.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092030 con 0x7f21c400a6b0 2021-03-09T10:45:04.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025c10 con 0x7f21c40037a0 2021-03-09T10:45:04.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00add0 con 0x7f21cc0138f0 2021-03-09T10:45:04.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:04.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:04.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:04.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:04.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.367+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:04.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:04.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:04.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:04.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:04.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc039e90 con 0x7f21cc0138f0 2021-03-09T10:45:04.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046250 con 0x7f21cc0138f0 2021-03-09T10:45:04.376 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.371+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14105 cs=0 l=1 rev1=1 rx=0x7f21cc04cf70 tx=0x7f21dc070e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:04.378 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:04.375+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48528 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:04.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.379+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0810d0 con 0x7f21cc0138f0 2021-03-09T10:45:04.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.379+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0810d0 con 0x7f21cc0138f0 2021-03-09T10:45:04.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.383+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:45:04.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.383+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:04.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.383+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:04.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:04.383+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:09.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.375+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:09.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.375+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:09.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.375+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14105 cs=0 l=1 rev1=1 rx=0x7f21cc04cf70 tx=0x7f21dc070e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:09.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.375+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14105 cs=0 l=1 rev1=1 rx=0x7f21cc04cf70 tx=0x7f21dc070e80).stop 2021-03-09T10:45:09.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.375+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:09.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.375+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=CLOSED pgs=14105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:09.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:09.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:09.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:09.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092030 con 0x7f21c400a6b0 2021-03-09T10:45:09.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025c10 con 0x7f21c40037a0 2021-03-09T10:45:09.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046720 con 0x7f21c4004620 2021-03-09T10:45:09.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:09.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:09.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:09.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:09.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:09.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:09.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:09.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:09.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:09.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0475c0 con 0x7f21c4004620 2021-03-09T10:45:09.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.379+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042cb0 con 0x7f21c4004620 2021-03-09T10:45:09.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.383+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046230 secure :-1 s=READY pgs=14117 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0091e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:09.388 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:09.387+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:09.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.391+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0007de0 con 0x7f21c4004620 2021-03-09T10:45:09.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.391+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0093ae0 con 0x7f21c4004620 2021-03-09T10:45:09.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.391+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:45:09.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.391+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:09.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.391+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:09.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:09.391+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:14.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc046230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc046230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046230 secure :-1 s=READY pgs=14117 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0091e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046230 secure :-1 s=READY pgs=14117 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0091e80).stop 2021-03-09T10:45:14.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc046230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046230 unknown :-1 s=CLOSED pgs=14117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:14.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c9a0 con 0x7f21c400a6b0 2021-03-09T10:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025c10 con 0x7f21c40037a0 2021-03-09T10:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042cb0 con 0x7f21cc0138f0 2021-03-09T10:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:14.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:14.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:14.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00aa90 con 0x7f21cc0138f0 2021-03-09T10:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.387+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc039e90 con 0x7f21cc0138f0 2021-03-09T10:45:14.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.391+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14126 cs=0 l=1 rev1=1 rx=0x7f21dc06dd90 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:14.396 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:14.395+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48604 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:14.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.395+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc009810 con 0x7f21cc0138f0 2021-03-09T10:45:14.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.395+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06dbd0 con 0x7f21cc0138f0 2021-03-09T10:45:14.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.399+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:45:14.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.399+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:14.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.399+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:14.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:14.399+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:19.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:45:19.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:19.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14126 cs=0 l=1 rev1=1 rx=0x7f21dc06dd90 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:19.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 secure :-1 s=READY pgs=14126 cs=0 l=1 rev1=1 rx=0x7f21dc06dd90 tx=0x7f21dc077de0).stop 2021-03-09T10:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0427c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0427c0 unknown :-1 s=CLOSED pgs=14126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:19.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:19.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0572f0 con 0x7f21c400a6b0 2021-03-09T10:45:19.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039e90 con 0x7f21c40037a0 2021-03-09T10:45:19.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00c9a0 con 0x7f21c4004620 2021-03-09T10:45:19.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:19.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:19.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:19.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:19.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:19.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:19.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:19.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:19.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.395+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:19.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.399+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046230 con 0x7f21c4004620 2021-03-09T10:45:19.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.399+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00aa90 con 0x7f21c4004620 2021-03-09T10:45:19.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.399+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091cb0 secure :-1 s=READY pgs=14136 cs=0 l=1 rev1=1 rx=0x7f21cc01d960 tx=0x7f21dc019220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:19.405 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:19.403+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48642 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:19.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:45:19.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06d9d0 con 0x7f21c4004620 2021-03-09T10:45:19.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21c4004620 2021-03-09T10:45:19.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.411+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:19.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.411+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:19.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:19.411+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:24.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:24.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:24.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091cb0 secure :-1 s=READY pgs=14136 cs=0 l=1 rev1=1 rx=0x7f21cc01d960 tx=0x7f21dc019220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:24.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091cb0 secure :-1 s=READY pgs=14136 cs=0 l=1 rev1=1 rx=0x7f21cc01d960 tx=0x7f21dc019220).stop 2021-03-09T10:45:24.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:24.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091cb0 unknown :-1 s=CLOSED pgs=14136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:24.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:24.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:24.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:24.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016040 con 0x7f21c400a6b0 2021-03-09T10:45:24.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0572f0 con 0x7f21c40037a0 2021-03-09T10:45:24.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039e90 con 0x7f21cc0138f0 2021-03-09T10:45:24.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:24.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:24.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:24.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:24.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.403+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:24.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00a9e0 con 0x7f21cc0138f0 2021-03-09T10:45:24.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0475c0 con 0x7f21cc0138f0 2021-03-09T10:45:24.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.407+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0463c0 secure :-1 s=READY pgs=14147 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e002b1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:24.413 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:24.411+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:24.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:45:24.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:45:24.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.415+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003ddb0 con 0x7f21cc0138f0 2021-03-09T10:45:24.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.415+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:24.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.415+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:24.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:24.415+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:28.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:28.509 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:29.416 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0463c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:29.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0463c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:29.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0463c0 secure :-1 s=READY pgs=14147 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e002b1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:29.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0463c0 secure :-1 s=READY pgs=14147 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e002b1d0).stop 2021-03-09T10:45:29.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0463c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:29.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0463c0 unknown :-1 s=CLOSED pgs=14147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:29.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:29.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:29.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:29.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039e90 con 0x7f21c400a6b0 2021-03-09T10:45:29.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00eb40 con 0x7f21c40037a0 2021-03-09T10:45:29.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:29.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ec80 con 0x7f21c4004620 2021-03-09T10:45:29.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:29.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:29.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.411+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:29.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:29.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:29.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:29.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:29.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:29.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091ac0 con 0x7f21c4004620 2021-03-09T10:45:29.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046230 con 0x7f21c4004620 2021-03-09T10:45:29.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.415+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14159 cs=0 l=1 rev1=1 rx=0x7f21dc069ca0 tx=0x7f21dc019530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:29.424 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:29.423+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48718 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:29.425 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0acbb0 con 0x7f21c4004620 2021-03-09T10:45:29.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.427+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:45:29.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:45:29.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.431+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:29.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.431+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:29.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:29.431+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:34.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:45:34.426 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:34.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14159 cs=0 l=1 rev1=1 rx=0x7f21dc069ca0 tx=0x7f21dc019530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:34.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14159 cs=0 l=1 rev1=1 rx=0x7f21dc069ca0 tx=0x7f21dc019530).stop 2021-03-09T10:45:34.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:34.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:34.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:34.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:34.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:34.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c400a6b0 2021-03-09T10:45:34.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00eaf0 con 0x7f21c40037a0 2021-03-09T10:45:34.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ec30 con 0x7f21cc0138f0 2021-03-09T10:45:34.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:34.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:34.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:34.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:34.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.423+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:34.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:34.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:34.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:34.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:34.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046230 con 0x7f21cc0138f0 2021-03-09T10:45:34.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091ac0 con 0x7f21cc0138f0 2021-03-09T10:45:34.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.427+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14169 cs=0 l=1 rev1=1 rx=0x7f21dc019370 tx=0x7f21dc0acbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:34.433 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:34.431+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:34.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0138f0 2021-03-09T10:45:34.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.435+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc062ec0 con 0x7f21cc0138f0 2021-03-09T10:45:34.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.435+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0138f0 2021-03-09T10:45:34.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.435+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:34.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.435+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:34.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:34.435+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:39.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:39.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:39.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14169 cs=0 l=1 rev1=1 rx=0x7f21dc019370 tx=0x7f21dc0acbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:39.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14169 cs=0 l=1 rev1=1 rx=0x7f21dc019370 tx=0x7f21dc0acbb0).stop 2021-03-09T10:45:39.435 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:39.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=CLOSED pgs=14169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:39.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:39.436 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:39.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:39.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:39.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:39.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0264e0 con 0x7f21c400a6b0 2021-03-09T10:45:39.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c40037a0 2021-03-09T10:45:39.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:39.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ec60 con 0x7f21c4004620 2021-03-09T10:45:39.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.431+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:39.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:39.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:39.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:39.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:39.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:39.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0463c0 con 0x7f21c4004620 2021-03-09T10:45:39.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046230 con 0x7f21c4004620 2021-03-09T10:45:39.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.435+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14179 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc018250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:39.442 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:39.439+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48794 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:39.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.443+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc065e40 con 0x7f21c4004620 2021-03-09T10:45:39.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.443+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc09a090 con 0x7f21c4004620 2021-03-09T10:45:39.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.443+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0acd70 con 0x7f21c4004620 2021-03-09T10:45:39.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.443+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:39.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.447+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:39.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:39.447+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:44.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:44.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:44.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14179 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc018250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:44.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14179 cs=0 l=1 rev1=1 rx=0x7f21dc07fdd0 tx=0x7f21dc018250).stop 2021-03-09T10:45:44.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:44.445 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:44.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:44.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:44.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:44.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08df40 con 0x7f21c400a6b0 2021-03-09T10:45:44.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c40037a0 2021-03-09T10:45:44.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21cc0138f0 2021-03-09T10:45:44.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:44.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:44.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:44.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:44.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.442+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:44.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:44.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:44.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:44.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:44.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046230 con 0x7f21cc0138f0 2021-03-09T10:45:44.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cf20 con 0x7f21cc0138f0 2021-03-09T10:45:44.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.446+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14195 cs=0 l=1 rev1=1 rx=0x7f21e408c040 tx=0x7f21e404f210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:44.451 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:44.450+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48832 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:44.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:45:44.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.454+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21cc0138f0 2021-03-09T10:45:44.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.454+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:45:44.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.454+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:44.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.454+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:44.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:44.454+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:49.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:45:49.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:49.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14195 cs=0 l=1 rev1=1 rx=0x7f21e408c040 tx=0x7f21e404f210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:49.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14195 cs=0 l=1 rev1=1 rx=0x7f21e408c040 tx=0x7f21e404f210).stop 2021-03-09T10:45:49.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:49.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=CLOSED pgs=14195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:49.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:49.454 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:49.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:49.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:45:49.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022800 con 0x7f21c40037a0 2021-03-09T10:45:49.455 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c4004620 2021-03-09T10:45:49.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:49.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:49.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:49.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:49.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.450+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:49.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:49.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:49.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:49.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:49.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08def0 con 0x7f21c4004620 2021-03-09T10:45:49.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046230 con 0x7f21c4004620 2021-03-09T10:45:49.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.454+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14202 cs=0 l=1 rev1=1 rx=0x7f21e4055570 tx=0x7f21e40a6510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:49.461 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:49.458+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48870 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:49.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.458+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40283a0 con 0x7f21c4004620 2021-03-09T10:45:49.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.462+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40283a0 con 0x7f21c4004620 2021-03-09T10:45:49.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.462+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008ac0 con 0x7f21c4004620 2021-03-09T10:45:49.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.462+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:49.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.462+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:49.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:49.462+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:54.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:54.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:54.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14202 cs=0 l=1 rev1=1 rx=0x7f21e4055570 tx=0x7f21e40a6510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:54.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14202 cs=0 l=1 rev1=1 rx=0x7f21e4055570 tx=0x7f21e40a6510).stop 2021-03-09T10:45:54.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:54.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:54.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:54.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:54.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:54.465 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:45:54.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c40037a0 2021-03-09T10:45:54.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022820 con 0x7f21cc0138f0 2021-03-09T10:45:54.466 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:54.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:54.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:54.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:54.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.458+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:54.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:54.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:54.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:54.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:54.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046230 con 0x7f21cc0138f0 2021-03-09T10:45:54.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0463c0 con 0x7f21cc0138f0 2021-03-09T10:45:54.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.462+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14211 cs=0 l=1 rev1=1 rx=0x7f21dc07bf90 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:54.471 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:54.470+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48909 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:54.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.470+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0b3ec0 con 0x7f21cc0138f0 2021-03-09T10:45:54.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0b3ec0 con 0x7f21cc0138f0 2021-03-09T10:45:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:45:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.478+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.478+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:54.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:54.478+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:45:58.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:58.565 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:45:59.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:45:59.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:59.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14211 cs=0 l=1 rev1=1 rx=0x7f21dc07bf90 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:59.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 secure :-1 s=READY pgs=14211 cs=0 l=1 rev1=1 rx=0x7f21dc07bf90 tx=0x7f21dc009880).stop 2021-03-09T10:45:59.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc091c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:59.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc091c50 unknown :-1 s=CLOSED pgs=14211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:59.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:59.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:59.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:59.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:45:59.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c40037a0 2021-03-09T10:45:59.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c4004620 2021-03-09T10:45:59.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:45:59.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:45:59.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:59.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:59.477 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:59.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:59.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:59.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:59.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:59.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041770 con 0x7f21c4004620 2021-03-09T10:45:59.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:45:59.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.470+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14222 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0007e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:59.481 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:45:59.478+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48947 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:45:59.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:45:59.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:45:59.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.482+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:45:59.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.482+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:45:59.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.482+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:45:59.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:45:59.482+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:04.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:04.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:04.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14222 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0007e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:04.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14222 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0007e50).stop 2021-03-09T10:46:04.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:04.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:04.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:04.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:46:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c40037a0 2021-03-09T10:46:04.485 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21cc0138f0 2021-03-09T10:46:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:04.486 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:04.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.478+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:04.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:04.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:04.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:04.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:04.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046580 con 0x7f21cc0138f0 2021-03-09T10:46:04.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046230 con 0x7f21cc0138f0 2021-03-09T10:46:04.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.482+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=14234 cs=0 l=1 rev1=1 rx=0x7f21cc032600 tx=0x7f21dc00afa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:04.490 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:04.486+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48985 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:04.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.490+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc0138f0 2021-03-09T10:46:04.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.494+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0191b0 con 0x7f21cc0138f0 2021-03-09T10:46:04.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.494+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:46:04.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.494+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:04.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.494+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:04.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:04.494+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:09.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:09.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:09.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=14234 cs=0 l=1 rev1=1 rx=0x7f21cc032600 tx=0x7f21dc00afa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:09.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=14234 cs=0 l=1 rev1=1 rx=0x7f21cc032600 tx=0x7f21dc00afa0).stop 2021-03-09T10:46:09.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:09.495 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=CLOSED pgs=14234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:09.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:09.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:09.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:09.496 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:46:09.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049b20 con 0x7f21c40037a0 2021-03-09T10:46:09.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f0c0 con 0x7f21c4004620 2021-03-09T10:46:09.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:09.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:09.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:09.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:09.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:09.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:09.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:09.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:09.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:09.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046230 con 0x7f21c4004620 2021-03-09T10:46:09.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.490+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046580 con 0x7f21c4004620 2021-03-09T10:46:09.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.494+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14243 cs=0 l=1 rev1=1 rx=0x7f21cc056920 tx=0x7f21dc0192b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:09.501 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:09.498+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49023 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:09.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.502+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005230 con 0x7f21c4004620 2021-03-09T10:46:09.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.506+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005230 con 0x7f21c4004620 2021-03-09T10:46:09.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.506+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0accb0 con 0x7f21c4004620 2021-03-09T10:46:09.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.506+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:09.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.510+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:09.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:09.510+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:14.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:14.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:14.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14243 cs=0 l=1 rev1=1 rx=0x7f21cc056920 tx=0x7f21dc0192b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:14.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14243 cs=0 l=1 rev1=1 rx=0x7f21cc056920 tx=0x7f21dc0192b0).stop 2021-03-09T10:46:14.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:14.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:14.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:14.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:14.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:14.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:46:14.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c40037a0 2021-03-09T10:46:14.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21cc0138f0 2021-03-09T10:46:14.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:14.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:14.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:14.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:14.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.498+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:14.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:14.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:14.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:14.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:14.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f0c0 con 0x7f21cc0138f0 2021-03-09T10:46:14.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08def0 con 0x7f21cc0138f0 2021-03-09T10:46:14.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.502+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=14254 cs=0 l=1 rev1=1 rx=0x7f21e407f740 tx=0x7f21e407f6b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:14.510 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:14.506+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49061 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:14.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.506+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f380 con 0x7f21cc0138f0 2021-03-09T10:46:14.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.510+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f380 con 0x7f21cc0138f0 2021-03-09T10:46:14.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.510+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:46:14.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.510+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:14.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.514+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:14.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:14.514+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:19.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:19.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:19.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=14254 cs=0 l=1 rev1=1 rx=0x7f21e407f740 tx=0x7f21e407f6b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:19.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 secure :-1 s=READY pgs=14254 cs=0 l=1 rev1=1 rx=0x7f21e407f740 tx=0x7f21e407f6b0).stop 2021-03-09T10:46:19.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc01bf90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:19.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc01bf90 unknown :-1 s=CLOSED pgs=14254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:19.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:19.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:19.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:19.514 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:46:19.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0892a0 con 0x7f21c40037a0 2021-03-09T10:46:19.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011a20 con 0x7f21c4004620 2021-03-09T10:46:19.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:19.515 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:19.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:19.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:19.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:19.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:19.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:19.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:19.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:19.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08dec0 con 0x7f21c4004620 2021-03-09T10:46:19.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041770 con 0x7f21c4004620 2021-03-09T10:46:19.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.510+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14264 cs=0 l=1 rev1=1 rx=0x7f21e0035ff0 tx=0x7f21e007d5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:19.522 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:19.518+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49097 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:19.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:46:19.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.518+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:46:19.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:46:19.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.522+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:19.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.522+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:19.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:19.522+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:24.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:24.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:24.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14264 cs=0 l=1 rev1=1 rx=0x7f21e0035ff0 tx=0x7f21e007d5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:24.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14264 cs=0 l=1 rev1=1 rx=0x7f21e0035ff0 tx=0x7f21e007d5b0).stop 2021-03-09T10:46:24.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:24.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:24.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:24.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:24.525 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:24.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092910 con 0x7f21c400a6b0 2021-03-09T10:46:24.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053b00 con 0x7f21c40037a0 2021-03-09T10:46:24.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21cc0138f0 2021-03-09T10:46:24.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:24.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:24.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:24.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:24.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:24.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:24.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:24.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:24.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:24.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011880 con 0x7f21cc0138f0 2021-03-09T10:46:24.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.522+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08dec0 con 0x7f21cc0138f0 2021-03-09T10:46:24.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.526+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e530 secure :-1 s=READY pgs=14277 cs=0 l=1 rev1=1 rx=0x7f21e0087d80 tx=0x7f21e002b160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:24.531 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:24.530+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49136 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:24.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:46:24.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0138f0 2021-03-09T10:46:24.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:46:24.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.538+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:24.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.538+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:24.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:24.538+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:28.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:28.614 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:29.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:29.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:29.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e530 secure :-1 s=READY pgs=14277 cs=0 l=1 rev1=1 rx=0x7f21e0087d80 tx=0x7f21e002b160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:29.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e530 secure :-1 s=READY pgs=14277 cs=0 l=1 rev1=1 rx=0x7f21e0087d80 tx=0x7f21e002b160).stop 2021-03-09T10:46:29.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04e530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:29.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e530 unknown :-1 s=CLOSED pgs=14277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:29.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:29.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:29.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:29.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0224a0 con 0x7f21c400a6b0 2021-03-09T10:46:29.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c40037a0 2021-03-09T10:46:29.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0928c0 con 0x7f21c4004620 2021-03-09T10:46:29.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:29.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:29.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:29.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:29.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:29.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:29.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.530+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:29.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.534+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:29.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.534+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:29.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.534+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011880 con 0x7f21c4004620 2021-03-09T10:46:29.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.534+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f0c0 con 0x7f21c4004620 2021-03-09T10:46:29.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.534+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14287 cs=0 l=1 rev1=1 rx=0x7f21cc056f90 tx=0x7f21e407f580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:29.540 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:29.538+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49173 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:29.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:46:29.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.542+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:46:29.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.546+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21c4004620 2021-03-09T10:46:29.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.546+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:29.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.546+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:29.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:29.546+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:34.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:34.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:34.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14287 cs=0 l=1 rev1=1 rx=0x7f21cc056f90 tx=0x7f21e407f580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:34.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14287 cs=0 l=1 rev1=1 rx=0x7f21cc056f90 tx=0x7f21e407f580).stop 2021-03-09T10:46:34.543 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:34.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:34.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:34.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:34.544 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:34.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056950 con 0x7f21c400a6b0 2021-03-09T10:46:34.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:34.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049be0 con 0x7f21c40037a0 2021-03-09T10:46:34.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022f20 con 0x7f21cc0138f0 2021-03-09T10:46:34.545 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:34.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:34.546 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:34.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.538+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:34.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:34.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:34.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:34.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:34.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011880 con 0x7f21cc0138f0 2021-03-09T10:46:34.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046230 con 0x7f21cc0138f0 2021-03-09T10:46:34.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.542+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e390 secure :-1 s=READY pgs=14301 cs=0 l=1 rev1=1 rx=0x7f21cc021460 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:34.550 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:34.546+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49212 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:34.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:46:34.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.550+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:46:34.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.554+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21cc0138f0 2021-03-09T10:46:34.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.554+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:34.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.554+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:34.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:34.554+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:39.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:39.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:39.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e390 secure :-1 s=READY pgs=14301 cs=0 l=1 rev1=1 rx=0x7f21cc021460 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:39.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e390 secure :-1 s=READY pgs=14301 cs=0 l=1 rev1=1 rx=0x7f21cc021460 tx=0x7f21dc077de0).stop 2021-03-09T10:46:39.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc04e390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:39.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc04e390 unknown :-1 s=CLOSED pgs=14301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:39.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:39.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:39.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:39.555 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc049be0 con 0x7f21c400a6b0 2021-03-09T10:46:39.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:39.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d7c0 con 0x7f21c40037a0 2021-03-09T10:46:39.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011880 con 0x7f21c4004620 2021-03-09T10:46:39.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:39.556 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:39.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:39.557 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:39.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:39.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:39.558 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:39.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:39.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056950 con 0x7f21c4004620 2021-03-09T10:46:39.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.549+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046230 con 0x7f21c4004620 2021-03-09T10:46:39.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.553+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14307 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc0b3190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:39.560 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:39.557+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49249 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:39.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4004620 2021-03-09T10:46:39.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005d60 con 0x7f21c4004620 2021-03-09T10:46:39.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.561+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4004620 2021-03-09T10:46:39.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.561+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:39.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.561+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:39.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:39.561+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:44.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:44.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:44.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14307 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc0b3190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:44.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 secure :-1 s=READY pgs=14307 cs=0 l=1 rev1=1 rx=0x7f21dc010d60 tx=0x7f21dc0b3190).stop 2021-03-09T10:46:44.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc00a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:44.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc00a9b0 unknown :-1 s=CLOSED pgs=14307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:44.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:44.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:44.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:44.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046420 con 0x7f21c400a6b0 2021-03-09T10:46:44.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022660 con 0x7f21c40037a0 2021-03-09T10:46:44.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0227a0 con 0x7f21cc0138f0 2021-03-09T10:46:44.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:44.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:44.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:44.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:44.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.557+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:44.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:44.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:44.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:44.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:44.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0228e0 con 0x7f21cc0138f0 2021-03-09T10:46:44.567 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f0c0 con 0x7f21cc0138f0 2021-03-09T10:46:44.568 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.561+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14320 cs=0 l=1 rev1=1 rx=0x7f21cc01abf0 tx=0x7f21e408a7c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:44.569 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:44.565+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49288 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:44.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21cc0138f0 2021-03-09T10:46:44.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21cc0138f0 2021-03-09T10:46:44.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.573+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21cc0138f0 2021-03-09T10:46:44.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.573+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:44.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:44.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:44.573+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:49.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14320 cs=0 l=1 rev1=1 rx=0x7f21cc01abf0 tx=0x7f21e408a7c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14320 cs=0 l=1 rev1=1 rx=0x7f21cc01abf0 tx=0x7f21e408a7c0).stop 2021-03-09T10:46:49.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:49.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:49.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:49.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:49.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:49.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21c400a6b0 2021-03-09T10:46:49.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f0c0 con 0x7f21c40037a0 2021-03-09T10:46:49.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a960 con 0x7f21c4004620 2021-03-09T10:46:49.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:49.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:49.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:49.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:49.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:49.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:49.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00aad0 con 0x7f21c4004620 2021-03-09T10:46:49.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056950 con 0x7f21c4004620 2021-03-09T10:46:49.577 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.569+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021600 secure :-1 s=READY pgs=14328 cs=0 l=1 rev1=1 rx=0x7f21dc0098e0 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:49.578 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:49.573+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49325 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:49.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068bc0 con 0x7f21c4004620 2021-03-09T10:46:49.579 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21c4004620 2021-03-09T10:46:49.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21c4004620 2021-03-09T10:46:49.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.577+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:49.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.581+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:49.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:49.581+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:54.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:54.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021600 secure :-1 s=READY pgs=14328 cs=0 l=1 rev1=1 rx=0x7f21dc0098e0 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021600 secure :-1 s=READY pgs=14328 cs=0 l=1 rev1=1 rx=0x7f21dc0098e0 tx=0x7f21dc02df50).stop 2021-03-09T10:46:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:54.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021600 unknown :-1 s=CLOSED pgs=14328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:54.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21c400a6b0 2021-03-09T10:46:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f0c0 con 0x7f21c40037a0 2021-03-09T10:46:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d70 con 0x7f21cc0138f0 2021-03-09T10:46:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:54.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:54.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:54.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:54.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:54.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:54.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04c0d0 con 0x7f21cc0138f0 2021-03-09T10:46:54.586 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.577+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00aad0 con 0x7f21cc0138f0 2021-03-09T10:46:54.587 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.581+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14341 cs=0 l=1 rev1=1 rx=0x7f21dc087690 tx=0x7f21dc068c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:54.588 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:54.585+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49364 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:54.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:46:54.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.585+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:46:54.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc005d20 con 0x7f21cc0138f0 2021-03-09T10:46:54.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.589+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:54.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.589+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:54.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:54.589+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:46:58.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:58.662 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:46:59.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:59.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:59.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14341 cs=0 l=1 rev1=1 rx=0x7f21dc087690 tx=0x7f21dc068c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:59.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14341 cs=0 l=1 rev1=1 rx=0x7f21dc087690 tx=0x7f21dc068c30).stop 2021-03-09T10:46:59.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:59.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:59.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:59.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:59.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0215a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:59.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21c400a6b0 2021-03-09T10:46:59.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d70 con 0x7f21c40037a0 2021-03-09T10:46:59.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c4004620 2021-03-09T10:46:59.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:46:59.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:59.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:46:59.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:59.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0215a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:59.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:59.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:59.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:59.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:59.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f0c0 con 0x7f21c4004620 2021-03-09T10:46:59.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04c0d0 con 0x7f21c4004620 2021-03-09T10:46:59.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.589+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0215a0 secure :-1 s=READY pgs=14352 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc068e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:59.603 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:46:59.597+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49401 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:46:59.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc096e10 con 0x7f21c4004620 2021-03-09T10:46:59.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc096e10 con 0x7f21c4004620 2021-03-09T10:46:59.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.601+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc029c20 con 0x7f21c4004620 2021-03-09T10:46:59.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.601+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:46:59.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.601+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:46:59.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:46:59.601+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:04.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0215a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:47:04.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0215a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:04.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0215a0 secure :-1 s=READY pgs=14352 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc068e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:04.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0215a0 secure :-1 s=READY pgs=14352 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc068e90).stop 2021-03-09T10:47:04.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0215a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:04.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0215a0 unknown :-1 s=CLOSED pgs=14352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:04.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:04.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:04.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:04.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465c0 con 0x7f21c400a6b0 2021-03-09T10:47:04.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042430 con 0x7f21c40037a0 2021-03-09T10:47:04.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0434f0 con 0x7f21cc0138f0 2021-03-09T10:47:04.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:04.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:04.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:04.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:04.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.597+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:04.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:04.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:04.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:04.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:04.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc04c270 con 0x7f21cc0138f0 2021-03-09T10:47:04.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f0c0 con 0x7f21cc0138f0 2021-03-09T10:47:04.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.601+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14361 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc068d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:04.608 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:04.605+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49440 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:04.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007e10 con 0x7f21cc0138f0 2021-03-09T10:47:04.609 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.605+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:47:04.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.609+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0accb0 con 0x7f21cc0138f0 2021-03-09T10:47:04.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.609+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:04.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.609+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:04.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:04.609+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:09.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:09.610 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:09.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14361 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc068d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:09.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14361 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc068d10).stop 2021-03-09T10:47:09.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:09.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:09.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:09.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:09.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:09.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e3a0 con 0x7f21c400a6b0 2021-03-09T10:47:09.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097740 con 0x7f21c40037a0 2021-03-09T10:47:09.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097880 con 0x7f21c4004620 2021-03-09T10:47:09.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:09.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:09.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:09.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:09.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:09.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:09.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:09.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:09.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:09.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac70 con 0x7f21c4004620 2021-03-09T10:47:09.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084090 con 0x7f21c4004620 2021-03-09T10:47:09.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.609+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021530 secure :-1 s=READY pgs=14371 cs=0 l=1 rev1=1 rx=0x7f21e0006330 tx=0x7f21e000bf80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:09.618 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:09.613+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49477 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:09.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.613+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000b3e0 con 0x7f21c4004620 2021-03-09T10:47:09.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.613+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0090080 con 0x7f21c4004620 2021-03-09T10:47:09.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.621+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:47:09.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.621+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:09.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.621+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:09.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:09.621+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:14.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:14.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:14.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021530 secure :-1 s=READY pgs=14371 cs=0 l=1 rev1=1 rx=0x7f21e0006330 tx=0x7f21e000bf80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:14.619 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021530 secure :-1 s=READY pgs=14371 cs=0 l=1 rev1=1 rx=0x7f21e0006330 tx=0x7f21e000bf80).stop 2021-03-09T10:47:14.620 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:14.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021530 unknown :-1 s=CLOSED pgs=14371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:14.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:14.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:14.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:14.621 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d70 con 0x7f21c400a6b0 2021-03-09T10:47:14.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0951f0 con 0x7f21c40037a0 2021-03-09T10:47:14.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095330 con 0x7f21cc0138f0 2021-03-09T10:47:14.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:14.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:14.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:14.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:14.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:14.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:14.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:14.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:14.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:14.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc097740 con 0x7f21cc0138f0 2021-03-09T10:47:14.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.617+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:47:14.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.621+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14383 cs=0 l=1 rev1=1 rx=0x7f21e0013ab0 tx=0x7f21e00138f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:14.627 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:14.625+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49516 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:14.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.625+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0091ec0 con 0x7f21cc0138f0 2021-03-09T10:47:14.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.625+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0091ec0 con 0x7f21cc0138f0 2021-03-09T10:47:14.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.629+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:47:14.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.629+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:14.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.629+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:14.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:14.629+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:19.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:47:19.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:19.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14383 cs=0 l=1 rev1=1 rx=0x7f21e0013ab0 tx=0x7f21e00138f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:19.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14383 cs=0 l=1 rev1=1 rx=0x7f21e0013ab0 tx=0x7f21e00138f0).stop 2021-03-09T10:47:19.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:19.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:19.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:19.630 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:19.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:19.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c400a6b0 2021-03-09T10:47:19.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092520 con 0x7f21c40037a0 2021-03-09T10:47:19.631 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a820 con 0x7f21c4004620 2021-03-09T10:47:19.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:19.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:19.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:19.632 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:19.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:19.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:19.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:19.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:19.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:19.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0951f0 con 0x7f21c4004620 2021-03-09T10:47:19.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc04c270 con 0x7f21c4004620 2021-03-09T10:47:19.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.625+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 secure :-1 s=READY pgs=14392 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc07cc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:19.636 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:19.633+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:19.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.633+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc021e20 con 0x7f21c4004620 2021-03-09T10:47:19.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.633+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21c4004620 2021-03-09T10:47:19.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:47:19.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.637+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:19.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.637+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:19.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:19.637+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:24.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:24.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:24.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 secure :-1 s=READY pgs=14392 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc07cc90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:24.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 secure :-1 s=READY pgs=14392 cs=0 l=1 rev1=1 rx=0x7f21dc0040b0 tx=0x7f21dc07cc90).stop 2021-03-09T10:47:24.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:24.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 unknown :-1 s=CLOSED pgs=14392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:24.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:24.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:24.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:24.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c400a6b0 2021-03-09T10:47:24.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011d70 con 0x7f21c40037a0 2021-03-09T10:47:24.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a820 con 0x7f21cc0138f0 2021-03-09T10:47:24.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:24.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:24.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:24.642 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:24.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:24.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:24.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:24.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:24.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:24.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00a960 con 0x7f21cc0138f0 2021-03-09T10:47:24.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0228e0 con 0x7f21cc0138f0 2021-03-09T10:47:24.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.637+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14406 cs=0 l=1 rev1=1 rx=0x7f21cc049cf0 tx=0x7f21e4033a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:24.646 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:24.641+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49592 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:24.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.645+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0138f0 2021-03-09T10:47:24.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.649+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0138f0 2021-03-09T10:47:24.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.653+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0138f0 2021-03-09T10:47:24.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.653+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:24.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:24.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:24.653+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:28.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:28.716 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:29.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:29.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:29.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14406 cs=0 l=1 rev1=1 rx=0x7f21cc049cf0 tx=0x7f21e4033a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:29.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14406 cs=0 l=1 rev1=1 rx=0x7f21cc049cf0 tx=0x7f21e4033a50).stop 2021-03-09T10:47:29.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:29.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:29.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:29.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:29.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:29.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c740 con 0x7f21c400a6b0 2021-03-09T10:47:29.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c40037a0 2021-03-09T10:47:29.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093d40 con 0x7f21c4004620 2021-03-09T10:47:29.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:29.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:29.651 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:29.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:29.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.644+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:29.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:29.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:29.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:29.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:29.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093e80 con 0x7f21c4004620 2021-03-09T10:47:29.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0951f0 con 0x7f21c4004620 2021-03-09T10:47:29.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.648+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 secure :-1 s=READY pgs=14416 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc018150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:29.656 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:29.652+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49629 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:29.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:47:29.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.652+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc013c50 con 0x7f21c4004620 2021-03-09T10:47:29.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:47:29.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.656+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:29.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.656+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:29.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:29.656+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:34.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:34.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:34.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 secure :-1 s=READY pgs=14416 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc018150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:34.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 secure :-1 s=READY pgs=14416 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc018150).stop 2021-03-09T10:47:34.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:34.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021510 unknown :-1 s=CLOSED pgs=14416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:34.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:34.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:34.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:34.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093d40 con 0x7f21c400a6b0 2021-03-09T10:47:34.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c740 con 0x7f21c40037a0 2021-03-09T10:47:34.660 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:47:34.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:34.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:34.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:34.661 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:34.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:34.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:34.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:34.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:34.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:34.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0951f0 con 0x7f21cc0138f0 2021-03-09T10:47:34.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00a960 con 0x7f21cc0138f0 2021-03-09T10:47:34.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.656+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14424 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4017d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:34.665 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:34.660+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49668 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:34.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4059000 con 0x7f21cc0138f0 2021-03-09T10:47:34.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21cc0138f0 2021-03-09T10:47:34.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.668+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0138f0 2021-03-09T10:47:34.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.668+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:34.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.668+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:34.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:34.668+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:39.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:39.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:39.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14424 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4017d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:39.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14424 cs=0 l=1 rev1=1 rx=0x7f21e400cd40 tx=0x7f21e4017d50).stop 2021-03-09T10:47:39.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:39.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:39.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:39.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0214f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:39.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:39.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046760 con 0x7f21c400a6b0 2021-03-09T10:47:39.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c40037a0 2021-03-09T10:47:39.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a060 con 0x7f21c4004620 2021-03-09T10:47:39.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:39.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:39.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:39.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:39.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:39.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:39.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057a20 con 0x7f21c4004620 2021-03-09T10:47:39.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0951f0 con 0x7f21c4004620 2021-03-09T10:47:39.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.664+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0214f0 secure :-1 s=READY pgs=14435 cs=0 l=1 rev1=1 rx=0x7f21cc049c10 tx=0x7f21e4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:39.675 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:39.672+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49705 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:39.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4015c80 con 0x7f21c4004620 2021-03-09T10:47:39.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:47:39.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.676+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4015c80 con 0x7f21c4004620 2021-03-09T10:47:39.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.676+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:39.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.676+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:39.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:39.676+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:44.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0214f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:47:44.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0214f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:44.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0214f0 secure :-1 s=READY pgs=14435 cs=0 l=1 rev1=1 rx=0x7f21cc049c10 tx=0x7f21e4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:44.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0214f0 secure :-1 s=READY pgs=14435 cs=0 l=1 rev1=1 rx=0x7f21cc049c10 tx=0x7f21e4005110).stop 2021-03-09T10:47:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0214f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0214f0 unknown :-1 s=CLOSED pgs=14435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:44.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0575b0 con 0x7f21c400a6b0 2021-03-09T10:47:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c8c0 con 0x7f21c40037a0 2021-03-09T10:47:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae70 con 0x7f21cc0138f0 2021-03-09T10:47:44.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:44.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:44.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:44.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:44.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:44.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:44.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:44.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:47:44.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.672+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057a20 con 0x7f21cc0138f0 2021-03-09T10:47:44.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.676+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14446 cs=0 l=1 rev1=1 rx=0x7f21e4009c40 tx=0x7f21e400ff50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:44.684 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:44.680+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49744 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:44.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.680+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a61c0 con 0x7f21cc0138f0 2021-03-09T10:47:44.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.680+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007d20 con 0x7f21cc0138f0 2021-03-09T10:47:44.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:47:44.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.684+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:44.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.684+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:44.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:44.684+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:49.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14446 cs=0 l=1 rev1=1 rx=0x7f21e4009c40 tx=0x7f21e400ff50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:49.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14446 cs=0 l=1 rev1=1 rx=0x7f21e4009c40 tx=0x7f21e400ff50).stop 2021-03-09T10:47:49.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:49.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0466b0 con 0x7f21c400a6b0 2021-03-09T10:47:49.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b310 con 0x7f21c40037a0 2021-03-09T10:47:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b450 con 0x7f21c4004620 2021-03-09T10:47:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:49.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:49.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:49.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:49.690 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:49.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:49.691 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:49.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:49.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:49.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093cf0 con 0x7f21c4004620 2021-03-09T10:47:49.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093e80 con 0x7f21c4004620 2021-03-09T10:47:49.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.684+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14456 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc09b760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:49.693 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:49.688+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49781 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:49.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc068e10 con 0x7f21c4004620 2021-03-09T10:47:49.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:47:49.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.692+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4004620 2021-03-09T10:47:49.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.692+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:49.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.692+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:49.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:49.692+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:54.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:54.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:54.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14456 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc09b760).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:54.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14456 cs=0 l=1 rev1=1 rx=0x7f21dc02e040 tx=0x7f21dc09b760).stop 2021-03-09T10:47:54.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:54.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:54.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:54.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc029cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:54.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:54.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057620 con 0x7f21c400a6b0 2021-03-09T10:47:54.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a1c0 con 0x7f21c40037a0 2021-03-09T10:47:54.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.688+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e80 con 0x7f21cc0138f0 2021-03-09T10:47:54.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:54.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:54.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:54.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:54.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc029cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:54.700 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:54.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:54.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:54.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:54.701 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0951f0 con 0x7f21cc0138f0 2021-03-09T10:47:54.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:47:54.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.692+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc029cd0 secure :-1 s=READY pgs=14467 cs=0 l=1 rev1=1 rx=0x7f21e407e4d0 tx=0x7f21e407e7d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:54.703 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:54.700+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49820 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:54.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017dc0 con 0x7f21cc0138f0 2021-03-09T10:47:54.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017dc0 con 0x7f21cc0138f0 2021-03-09T10:47:54.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21cc0138f0 2021-03-09T10:47:54.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.700+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:54.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.704+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:54.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:54.704+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:47:58.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:58.764 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:47:59.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc029cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc029cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc029cd0 secure :-1 s=READY pgs=14467 cs=0 l=1 rev1=1 rx=0x7f21e407e4d0 tx=0x7f21e407e7d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:59.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc029cd0 secure :-1 s=READY pgs=14467 cs=0 l=1 rev1=1 rx=0x7f21e407e4d0 tx=0x7f21e407e7d0).stop 2021-03-09T10:47:59.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc029cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:59.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc029cd0 unknown :-1 s=CLOSED pgs=14467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:59.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:59.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:59.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:59.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a1c0 con 0x7f21c400a6b0 2021-03-09T10:47:59.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e80 con 0x7f21c40037a0 2021-03-09T10:47:59.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c4004620 2021-03-09T10:47:59.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:47:59.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:59.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:47:59.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:59.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:59.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:59.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025e40 con 0x7f21c4004620 2021-03-09T10:47:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0951f0 con 0x7f21c4004620 2021-03-09T10:47:59.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.700+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14478 cs=0 l=1 rev1=1 rx=0x7f21e40245b0 tx=0x7f21e400ff50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:59.711 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:47:59.708+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49857 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:47:59.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40241e0 con 0x7f21c4004620 2021-03-09T10:47:59.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40241e0 con 0x7f21c4004620 2021-03-09T10:47:59.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.708+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4180 con 0x7f21c4004620 2021-03-09T10:47:59.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.708+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:47:59.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.712+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:47:59.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:47:59.712+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:04.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:04.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:04.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14478 cs=0 l=1 rev1=1 rx=0x7f21e40245b0 tx=0x7f21e400ff50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:04.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14478 cs=0 l=1 rev1=1 rx=0x7f21e40245b0 tx=0x7f21e400ff50).stop 2021-03-09T10:48:04.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:04.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:04.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:04.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:04.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:04.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d410 con 0x7f21c400a6b0 2021-03-09T10:48:04.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e80 con 0x7f21c40037a0 2021-03-09T10:48:04.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:48:04.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:04.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:04.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:04.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:04.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:04.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:04.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:04.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:04.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:04.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0951f0 con 0x7f21cc0138f0 2021-03-09T10:48:04.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093cf0 con 0x7f21cc0138f0 2021-03-09T10:48:04.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.712+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14488 cs=0 l=1 rev1=1 rx=0x7f21cc089740 tx=0x7f21dc009810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:04.720 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:04.716+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49896 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:04.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.716+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:48:04.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.716+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc019e80 con 0x7f21cc0138f0 2021-03-09T10:48:04.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.720+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:48:04.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.720+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:04.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.720+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:04.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:04.720+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:09.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:09.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:09.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14488 cs=0 l=1 rev1=1 rx=0x7f21cc089740 tx=0x7f21dc009810).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:09.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14488 cs=0 l=1 rev1=1 rx=0x7f21cc089740 tx=0x7f21dc009810).stop 2021-03-09T10:48:09.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:09.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:09.722 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:09.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:09.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:09.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056f90 con 0x7f21c400a6b0 2021-03-09T10:48:09.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c40037a0 2021-03-09T10:48:09.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b10 con 0x7f21c4004620 2021-03-09T10:48:09.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:09.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:09.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:09.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:09.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:09.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:09.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:09.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:09.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:09.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc093c50 con 0x7f21c4004620 2021-03-09T10:48:09.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc097740 con 0x7f21c4004620 2021-03-09T10:48:09.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.720+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14499 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e0003740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:09.729 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:09.724+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49933 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:09.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21c4004620 2021-03-09T10:48:09.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21c4004620 2021-03-09T10:48:09.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.728+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:48:09.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.728+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:09.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.728+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:09.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:09.728+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:14.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:14.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:14.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14499 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e0003740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:14.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14499 cs=0 l=1 rev1=1 rx=0x7f21e00a1010 tx=0x7f21e0003740).stop 2021-03-09T10:48:14.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:14.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:14.732 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:14.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:14.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:14.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b10 con 0x7f21c400a6b0 2021-03-09T10:48:14.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097900 con 0x7f21c40037a0 2021-03-09T10:48:14.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:48:14.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:14.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:14.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:14.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.724+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:14.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:14.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:14.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:14.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:14.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:14.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a010 con 0x7f21cc0138f0 2021-03-09T10:48:14.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0951f0 con 0x7f21cc0138f0 2021-03-09T10:48:14.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.728+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14510 cs=0 l=1 rev1=1 rx=0x7f21dc005e00 tx=0x7f21dc005cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:14.738 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:14.732+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49972 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:14.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.732+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc005e70 con 0x7f21cc0138f0 2021-03-09T10:48:14.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.732+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc005e70 con 0x7f21cc0138f0 2021-03-09T10:48:14.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.740+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:48:14.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.740+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:14.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.740+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:14.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:14.740+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:19.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:48:19.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:19.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14510 cs=0 l=1 rev1=1 rx=0x7f21dc005e00 tx=0x7f21dc005cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:19.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 secure :-1 s=READY pgs=14510 cs=0 l=1 rev1=1 rx=0x7f21dc005e00 tx=0x7f21dc005cb0).stop 2021-03-09T10:48:19.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc011880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:19.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc011880 unknown :-1 s=CLOSED pgs=14510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:19.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:19.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:19.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:19.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:19.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:19.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b10 con 0x7f21c400a6b0 2021-03-09T10:48:19.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c230 con 0x7f21c40037a0 2021-03-09T10:48:19.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac70 con 0x7f21c4004620 2021-03-09T10:48:19.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:19.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:19.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:19.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:19.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:19.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:19.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:19.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0978e0 con 0x7f21c4004620 2021-03-09T10:48:19.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025e40 con 0x7f21c4004620 2021-03-09T10:48:19.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.736+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14521 cs=0 l=1 rev1=1 rx=0x7f21e4065340 tx=0x7f21e40a6230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:19.747 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:19.744+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50009 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:19.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.744+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4065370 con 0x7f21c4004620 2021-03-09T10:48:19.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.748+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4065370 con 0x7f21c4004620 2021-03-09T10:48:19.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.752+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:48:19.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.752+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:19.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.752+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:19.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:19.752+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:24.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:24.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:24.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14521 cs=0 l=1 rev1=1 rx=0x7f21e4065340 tx=0x7f21e40a6230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:24.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14521 cs=0 l=1 rev1=1 rx=0x7f21e4065340 tx=0x7f21e40a6230).stop 2021-03-09T10:48:24.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:24.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:24.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:24.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:24.750 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:24.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c230 con 0x7f21c400a6b0 2021-03-09T10:48:24.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093b10 con 0x7f21c40037a0 2021-03-09T10:48:24.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f90 con 0x7f21cc0138f0 2021-03-09T10:48:24.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:24.751 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:24.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:24.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:24.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:24.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:24.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:24.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:24.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:24.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:48:24.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc093c50 con 0x7f21cc0138f0 2021-03-09T10:48:24.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.743+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14531 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e001f6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:24.757 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:24.751+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50047 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:24.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.751+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0029110 con 0x7f21cc0138f0 2021-03-09T10:48:24.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.751+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:48:24.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.755+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:48:24.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.755+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:24.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.755+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:24.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:24.755+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:28.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:28.811 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:29.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:29.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:29.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14531 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e001f6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:29.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14531 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e001f6c0).stop 2021-03-09T10:48:29.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:29.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:29.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:29.761 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:29.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:29.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c230 con 0x7f21c400a6b0 2021-03-09T10:48:29.762 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc016f00 con 0x7f21c40037a0 2021-03-09T10:48:29.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f90 con 0x7f21c4004620 2021-03-09T10:48:29.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:29.763 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:29.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:29.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:29.764 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:29.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:29.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:29.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:29.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:29.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0468c0 con 0x7f21c4004620 2021-03-09T10:48:29.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0978e0 con 0x7f21c4004620 2021-03-09T10:48:29.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.755+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14542 cs=0 l=1 rev1=1 rx=0x7f21cc0953b0 tx=0x7f21e4053ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:29.768 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:29.759+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50085 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:29.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.759+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4004620 2021-03-09T10:48:29.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.759+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1280 con 0x7f21c4004620 2021-03-09T10:48:29.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.763+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21c4004620 2021-03-09T10:48:29.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.763+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:29.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.767+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:29.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:29.767+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:34.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:34.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:34.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14542 cs=0 l=1 rev1=1 rx=0x7f21cc0953b0 tx=0x7f21e4053ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:34.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14542 cs=0 l=1 rev1=1 rx=0x7f21cc0953b0 tx=0x7f21e4053ed0).stop 2021-03-09T10:48:34.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:34.769 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:34.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:34.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:34.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:34.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.763+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c230 con 0x7f21c400a6b0 2021-03-09T10:48:34.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f90 con 0x7f21c40037a0 2021-03-09T10:48:34.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0131a0 con 0x7f21cc0138f0 2021-03-09T10:48:34.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:34.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:34.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:34.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:34.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:34.773 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:34.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:34.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:34.774 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:34.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025d80 con 0x7f21cc0138f0 2021-03-09T10:48:34.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ac70 con 0x7f21cc0138f0 2021-03-09T10:48:34.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.767+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14552 cs=0 l=1 rev1=1 rx=0x7f21e001f880 tx=0x7f21e007d540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:34.776 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:34.775+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50123 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:34.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21cc0138f0 2021-03-09T10:48:34.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002b160 con 0x7f21cc0138f0 2021-03-09T10:48:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.779+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:48:34.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.779+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:34.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.779+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:34.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:34.779+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:39.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:39.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:39.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14552 cs=0 l=1 rev1=1 rx=0x7f21e001f880 tx=0x7f21e007d540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:39.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14552 cs=0 l=1 rev1=1 rx=0x7f21e001f880 tx=0x7f21e007d540).stop 2021-03-09T10:48:39.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:39.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:39.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:39.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:39.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:39.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c230 con 0x7f21c400a6b0 2021-03-09T10:48:39.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f90 con 0x7f21c40037a0 2021-03-09T10:48:39.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029750 con 0x7f21c4004620 2021-03-09T10:48:39.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:39.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:39.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:39.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:39.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.775+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:39.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:39.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:39.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:39.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:39.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0298c0 con 0x7f21c4004620 2021-03-09T10:48:39.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0468c0 con 0x7f21c4004620 2021-03-09T10:48:39.783 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.779+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14563 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e40094f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:39.784 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:39.783+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50161 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:39.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:48:39.784 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d30 con 0x7f21c4004620 2021-03-09T10:48:39.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.787+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e407e190 con 0x7f21c4004620 2021-03-09T10:48:39.787 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.787+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:39.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.787+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:39.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:39.787+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:44.786 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:44.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:44.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14563 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e40094f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:44.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14563 cs=0 l=1 rev1=1 rx=0x7f21e409d0b0 tx=0x7f21e40094f0).stop 2021-03-09T10:48:44.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:44.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:44.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:44.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:44.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:44.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e250 con 0x7f21c400a6b0 2021-03-09T10:48:44.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01e390 con 0x7f21c40037a0 2021-03-09T10:48:44.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0562d0 con 0x7f21cc0138f0 2021-03-09T10:48:44.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:44.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:44.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:44.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:44.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.783+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:44.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:44.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:44.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:44.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:44.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056440 con 0x7f21cc0138f0 2021-03-09T10:48:44.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a010 con 0x7f21cc0138f0 2021-03-09T10:48:44.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.787+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14578 cs=0 l=1 rev1=1 rx=0x7f21dc0ab5e0 tx=0x7f21dc0ab4b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:44.879 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:44.795+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50199 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:44.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:48:44.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.799+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:48:44.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.799+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:48:44.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.799+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:44.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.803+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:44.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:44.803+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:49.796 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:48:49.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:49.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14578 cs=0 l=1 rev1=1 rx=0x7f21dc0ab5e0 tx=0x7f21dc0ab4b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:49.797 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14578 cs=0 l=1 rev1=1 rx=0x7f21dc0ab5e0 tx=0x7f21dc0ab4b0).stop 2021-03-09T10:48:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:49.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:49.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:49.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:49.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:49.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:49.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056770 con 0x7f21c400a6b0 2021-03-09T10:48:49.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a1b0 con 0x7f21c40037a0 2021-03-09T10:48:49.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056280 con 0x7f21c4004620 2021-03-09T10:48:49.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:49.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:49.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.795+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:49.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:49.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:49.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:49.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:49.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01e250 con 0x7f21c4004620 2021-03-09T10:48:49.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0298c0 con 0x7f21c4004620 2021-03-09T10:48:49.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.799+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14590 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:49.804 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:49.803+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50237 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:49.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:48:49.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.807+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:48:49.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.811+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007be0 con 0x7f21c4004620 2021-03-09T10:48:49.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.811+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:49.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.811+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:49.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:49.811+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:54.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:54.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:54.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14590 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:54.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14590 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e4005110).stop 2021-03-09T10:48:54.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:54.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:54.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:54.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:54.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:54.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:54.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091990 con 0x7f21c400a6b0 2021-03-09T10:48:54.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:54.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:54.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:54.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c40037a0 2021-03-09T10:48:54.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056770 con 0x7f21cc0138f0 2021-03-09T10:48:54.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.803+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:54.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:54.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:54.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:54.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:54.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a1b0 con 0x7f21cc0138f0 2021-03-09T10:48:54.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056440 con 0x7f21cc0138f0 2021-03-09T10:48:54.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.807+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14598 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc010d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:54.814 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:54.811+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50275 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:54.815 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:48:54.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:48:54.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.819+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:48:54.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.819+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:54.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.819+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:54.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:54.819+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:48:58.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:58.923 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:48:59.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:59.816 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:59.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14598 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc010d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:59.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14598 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc010d90).stop 2021-03-09T10:48:59.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:59.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:59.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:59.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:59.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:59.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c400a6b0 2021-03-09T10:48:59.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091950 con 0x7f21c40037a0 2021-03-09T10:48:59.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056280 con 0x7f21c4004620 2021-03-09T10:48:59.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:48:59.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:59.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:59.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:48:59.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:59.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:59.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:59.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:59.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:59.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0563c0 con 0x7f21c4004620 2021-03-09T10:48:59.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025d80 con 0x7f21c4004620 2021-03-09T10:48:59.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.815+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14606 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e002b1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:59.824 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:48:59.823+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50313 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:48:59.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:48:59.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:48:59.827 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.827+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:48:59.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.827+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:48:59.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.827+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:48:59.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:48:59.827+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:04.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:04.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:04.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14606 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e002b1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:04.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14606 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e002b1d0).stop 2021-03-09T10:49:04.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:04.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:04.826 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:04.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:04.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:04.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc082db0 con 0x7f21c400a6b0 2021-03-09T10:49:04.828 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:04.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056230 con 0x7f21c40037a0 2021-03-09T10:49:04.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056550 con 0x7f21cc0138f0 2021-03-09T10:49:04.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:04.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:04.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:04.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:04.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:04.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:04.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:04.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:04.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0566c0 con 0x7f21cc0138f0 2021-03-09T10:49:04.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.823+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a1b0 con 0x7f21cc0138f0 2021-03-09T10:49:04.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.827+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14616 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc009880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:04.833 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:04.831+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50351 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:04.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc033420 con 0x7f21cc0138f0 2021-03-09T10:49:04.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc033420 con 0x7f21cc0138f0 2021-03-09T10:49:04.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.835+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:49:04.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.835+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:04.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.835+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:04.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:04.835+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:09.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:09.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:09.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14616 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc009880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:09.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14616 cs=0 l=1 rev1=1 rx=0x7f21dc009960 tx=0x7f21dc009880).stop 2021-03-09T10:49:09.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:09.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:09.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:09.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:09.836 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:09.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041ab0 con 0x7f21c400a6b0 2021-03-09T10:49:09.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a010 con 0x7f21c40037a0 2021-03-09T10:49:09.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a150 con 0x7f21c4004620 2021-03-09T10:49:09.837 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:09.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:09.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:09.838 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.831+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:09.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:09.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:09.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:09.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:09.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:09.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025d10 con 0x7f21c4004620 2021-03-09T10:49:09.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01e250 con 0x7f21c4004620 2021-03-09T10:49:09.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.835+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14627 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e40552f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:09.844 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:09.843+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50389 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:09.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21c4004620 2021-03-09T10:49:09.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.843+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4007be0 con 0x7f21c4004620 2021-03-09T10:49:09.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.847+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007be0 con 0x7f21c4004620 2021-03-09T10:49:09.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.847+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:09.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.847+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:09.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:09.847+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:14.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:49:14.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:14.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14627 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e40552f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:14.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14627 cs=0 l=1 rev1=1 rx=0x7f21e40074a0 tx=0x7f21e40552f0).stop 2021-03-09T10:49:14.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:14.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:14.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:14.847 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:14.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:14.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026d20 con 0x7f21c400a6b0 2021-03-09T10:49:14.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:14.848 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a010 con 0x7f21c40037a0 2021-03-09T10:49:14.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a150 con 0x7f21cc0138f0 2021-03-09T10:49:14.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:14.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:14.849 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:14.850 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:14.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:14.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:14.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:14.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:14.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cb60 con 0x7f21cc0138f0 2021-03-09T10:49:14.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.842+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025d10 con 0x7f21cc0138f0 2021-03-09T10:49:14.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.846+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14638 cs=0 l=1 rev1=1 rx=0x7f21e400db60 tx=0x7f21e400de60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:14.853 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:14.850+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50427 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:14.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4008ed0 con 0x7f21cc0138f0 2021-03-09T10:49:14.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:49:14.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.854+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21cc0138f0 2021-03-09T10:49:14.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.854+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:14.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.854+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:14.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:14.854+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:19.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:19.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:19.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14638 cs=0 l=1 rev1=1 rx=0x7f21e400db60 tx=0x7f21e400de60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:19.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14638 cs=0 l=1 rev1=1 rx=0x7f21e400db60 tx=0x7f21e400de60).stop 2021-03-09T10:49:19.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:19.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:19.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:19.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:19.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:19.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e5a0 con 0x7f21c400a6b0 2021-03-09T10:49:19.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ccf0 con 0x7f21c40037a0 2021-03-09T10:49:19.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.850+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ce30 con 0x7f21c4004620 2021-03-09T10:49:19.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:19.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:19.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:19.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:19.859 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:19.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:19.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:19.860 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:19.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:19.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026cf0 con 0x7f21c4004620 2021-03-09T10:49:19.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0563c0 con 0x7f21c4004620 2021-03-09T10:49:19.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.854+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14649 cs=0 l=1 rev1=1 rx=0x7f21e0004fd0 tx=0x7f21e0090e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:19.863 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:19.862+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50465 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:19.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:49:19.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:49:19.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.866+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:49:19.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.866+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:19.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.866+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:19.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:19.866+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:24.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:24.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:24.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14649 cs=0 l=1 rev1=1 rx=0x7f21e0004fd0 tx=0x7f21e0090e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:24.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14649 cs=0 l=1 rev1=1 rx=0x7f21e0004fd0 tx=0x7f21e0090e70).stop 2021-03-09T10:49:24.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:24.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:24.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:24.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:24.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:24.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043020 con 0x7f21c400a6b0 2021-03-09T10:49:24.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c620 con 0x7f21c40037a0 2021-03-09T10:49:24.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ccf0 con 0x7f21cc0138f0 2021-03-09T10:49:24.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:24.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:24.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:24.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:24.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:24.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:24.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:24.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:24.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:24.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01ce30 con 0x7f21cc0138f0 2021-03-09T10:49:24.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc026cf0 con 0x7f21cc0138f0 2021-03-09T10:49:24.870 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.862+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14659 cs=0 l=1 rev1=1 rx=0x7f21e0059d90 tx=0x7f21e00a6270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:24.872 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:24.870+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50504 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:24.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00142e0 con 0x7f21cc0138f0 2021-03-09T10:49:24.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:49:24.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.874+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:49:24.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.874+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:24.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.874+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:24.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:24.874+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:28.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:28.976 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:29.873 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:29.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:29.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14659 cs=0 l=1 rev1=1 rx=0x7f21e0059d90 tx=0x7f21e00a6270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:29.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 secure :-1 s=READY pgs=14659 cs=0 l=1 rev1=1 rx=0x7f21e0059d90 tx=0x7f21e00a6270).stop 2021-03-09T10:49:29.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc021ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:29.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc021ac0 unknown :-1 s=CLOSED pgs=14659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:29.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:29.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:29.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:29.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ccf0 con 0x7f21c400a6b0 2021-03-09T10:49:29.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec80 con 0x7f21c40037a0 2021-03-09T10:49:29.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025eb0 con 0x7f21c4004620 2021-03-09T10:49:29.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:29.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:29.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.870+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:29.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:29.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:29.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:29.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:29.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:29.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:29.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc026b00 con 0x7f21c4004620 2021-03-09T10:49:29.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01ce30 con 0x7f21c4004620 2021-03-09T10:49:29.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.874+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14670 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0029180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:29.883 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50541 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:29.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0076680 con 0x7f21c4004620 2021-03-09T10:49:29.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00764a0 con 0x7f21c4004620 2021-03-09T10:49:29.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21c4004620 2021-03-09T10:49:29.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:29.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:29.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:29.882+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:34.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:34.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:34.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14670 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0029180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:34.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14670 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0029180).stop 2021-03-09T10:49:34.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:34.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:34.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:34.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:34.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:34.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ccf0 con 0x7f21c400a6b0 2021-03-09T10:49:34.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01ce30 con 0x7f21c40037a0 2021-03-09T10:49:34.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025d10 con 0x7f21cc0138f0 2021-03-09T10:49:34.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:34.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:34.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:34.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:34.889 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:34.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:34.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:34.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:34.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:34.890 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025e50 con 0x7f21cc0138f0 2021-03-09T10:49:34.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.882+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cb60 con 0x7f21cc0138f0 2021-03-09T10:49:34.891 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.886+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026ab0 secure :-1 s=READY pgs=14680 cs=0 l=1 rev1=1 rx=0x7f21e407e350 tx=0x7f21e4053f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:34.893 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:34.890+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50578 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:34.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4028120 con 0x7f21cc0138f0 2021-03-09T10:49:34.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f170 con 0x7f21cc0138f0 2021-03-09T10:49:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.894+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e400dbd0 con 0x7f21cc0138f0 2021-03-09T10:49:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.894+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.894+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:34.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:34.894+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:39.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:39.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:39.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026ab0 secure :-1 s=READY pgs=14680 cs=0 l=1 rev1=1 rx=0x7f21e407e350 tx=0x7f21e4053f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:39.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026ab0 secure :-1 s=READY pgs=14680 cs=0 l=1 rev1=1 rx=0x7f21e407e350 tx=0x7f21e4053f80).stop 2021-03-09T10:49:39.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:39.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026ab0 unknown :-1 s=CLOSED pgs=14680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:39.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:39.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:39.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:39.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025d10 con 0x7f21c400a6b0 2021-03-09T10:49:39.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cb10 con 0x7f21c40037a0 2021-03-09T10:49:39.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cc50 con 0x7f21c4004620 2021-03-09T10:49:39.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:39.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:39.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:39.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.890+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:39.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.894+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.894+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.894+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:39.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.894+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.894+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01cd90 con 0x7f21c4004620 2021-03-09T10:49:39.900 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.898+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025e50 con 0x7f21c4004620 2021-03-09T10:49:39.901 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.898+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14697 cs=0 l=1 rev1=1 rx=0x7f21e4036560 tx=0x7f21e407f1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:39.902 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:39.902+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50617 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:39.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027d20 con 0x7f21c4004620 2021-03-09T10:49:39.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027d20 con 0x7f21c4004620 2021-03-09T10:49:39.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4016080 con 0x7f21c4004620 2021-03-09T10:49:39.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.902+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:39.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:39.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:39.906+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:44.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:44.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14697 cs=0 l=1 rev1=1 rx=0x7f21e4036560 tx=0x7f21e407f1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14697 cs=0 l=1 rev1=1 rx=0x7f21e4036560 tx=0x7f21e407f1e0).stop 2021-03-09T10:49:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:44.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:44.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:44.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012ef0 con 0x7f21c400a6b0 2021-03-09T10:49:44.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc013030 con 0x7f21c40037a0 2021-03-09T10:49:44.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0219a0 con 0x7f21cc0138f0 2021-03-09T10:49:44.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:44.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.902+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:44.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:44.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:44.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:44.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:44.910 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021ae0 con 0x7f21cc0138f0 2021-03-09T10:49:44.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0566c0 con 0x7f21cc0138f0 2021-03-09T10:49:44.911 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.906+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026a10 secure :-1 s=READY pgs=14709 cs=0 l=1 rev1=1 rx=0x7f21cc021970 tx=0x7f21dc018150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:44.912 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:44.910+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50654 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:44.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc09a080 con 0x7f21cc0138f0 2021-03-09T10:49:44.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc017000 con 0x7f21cc0138f0 2021-03-09T10:49:44.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.918+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:49:44.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.918+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:44.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:44.922 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:44.918+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:49.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:49.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:49.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026a10 secure :-1 s=READY pgs=14709 cs=0 l=1 rev1=1 rx=0x7f21cc021970 tx=0x7f21dc018150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:49.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026a10 secure :-1 s=READY pgs=14709 cs=0 l=1 rev1=1 rx=0x7f21cc021970 tx=0x7f21dc018150).stop 2021-03-09T10:49:49.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:49.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026a10 unknown :-1 s=CLOSED pgs=14709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:49.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:49.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:49.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:49.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d6e0 con 0x7f21c400a6b0 2021-03-09T10:49:49.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057530 con 0x7f21c40037a0 2021-03-09T10:49:49.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021c70 con 0x7f21c4004620 2021-03-09T10:49:49.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:49.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:49.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:49.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.910+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:49.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:49.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:49.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:49.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:49.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:49.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021db0 con 0x7f21c4004620 2021-03-09T10:49:49.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.914+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01cd90 con 0x7f21c4004620 2021-03-09T10:49:49.921 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.918+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14716 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407e190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:49.923 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:49.922+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50693 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:49.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e402d640 con 0x7f21c4004620 2021-03-09T10:49:49.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402d640 con 0x7f21c4004620 2021-03-09T10:49:49.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.926+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4007160 con 0x7f21c4004620 2021-03-09T10:49:49.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.926+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:49.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:49.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:49.926+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:54.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:54.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:54.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14716 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407e190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:54.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14716 cs=0 l=1 rev1=1 rx=0x7f21e4021040 tx=0x7f21e407e190).stop 2021-03-09T10:49:54.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:54.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:54.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:54.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:54.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057510 con 0x7f21c400a6b0 2021-03-09T10:49:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057650 con 0x7f21c40037a0 2021-03-09T10:49:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f40 con 0x7f21cc0138f0 2021-03-09T10:49:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:54.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:54.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:54.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:54.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:54.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:54.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:54.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:54.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:54.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0942e0 con 0x7f21cc0138f0 2021-03-09T10:49:54.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.922+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021ae0 con 0x7f21cc0138f0 2021-03-09T10:49:54.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.926+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026970 secure :-1 s=READY pgs=14723 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0abee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:54.933 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:54.930+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50730 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:54.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069ae0 con 0x7f21cc0138f0 2021-03-09T10:49:54.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069ae0 con 0x7f21cc0138f0 2021-03-09T10:49:54.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.934+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:49:54.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.934+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:54.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:54.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:54.934+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:49:59.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:59.023 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:49:59.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:49:59.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:59.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026970 secure :-1 s=READY pgs=14723 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0abee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:59.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026970 secure :-1 s=READY pgs=14723 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0abee0).stop 2021-03-09T10:49:59.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc026970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:59.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc026970 unknown :-1 s=CLOSED pgs=14723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:59.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:59.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:59.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:59.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021ae0 con 0x7f21c400a6b0 2021-03-09T10:49:59.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091f20 con 0x7f21c40037a0 2021-03-09T10:49:59.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc092060 con 0x7f21c4004620 2021-03-09T10:49:59.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:49:59.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:59.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:49:59.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.930+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:59.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:59.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:59.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:59.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:59.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:59.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0574c0 con 0x7f21c4004620 2021-03-09T10:49:59.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.934+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:49:59.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.938+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14736 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e00a6200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:59.942 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:49:59.938+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50769 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:49:59.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0007de0 con 0x7f21c4004620 2021-03-09T10:49:59.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:49:59.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.946+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:49:59.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.946+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:49:59.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:49:59.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:49:59.950+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:04.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:04.944 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:04.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14736 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e00a6200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:04.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14736 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e00a6200).stop 2021-03-09T10:50:04.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:04.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:04.945 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:04.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:04.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:04.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057650 con 0x7f21c400a6b0 2021-03-09T10:50:04.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01b9c0 con 0x7f21c40037a0 2021-03-09T10:50:04.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021ae0 con 0x7f21cc0138f0 2021-03-09T10:50:04.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:04.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:04.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:04.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:04.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:04.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:04.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:04.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:04.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:04.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017470 con 0x7f21cc0138f0 2021-03-09T10:50:04.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0574c0 con 0x7f21cc0138f0 2021-03-09T10:50:04.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.942+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268d0 secure :-1 s=READY pgs=14744 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0075c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:04.952 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:04.950+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50806 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:04.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fc40 con 0x7f21cc0138f0 2021-03-09T10:50:04.953 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.950+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0004070 con 0x7f21cc0138f0 2021-03-09T10:50:04.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.954+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003c850 con 0x7f21cc0138f0 2021-03-09T10:50:04.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.954+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:04.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:04.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:04.954+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:09.954 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:09.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:09.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268d0 secure :-1 s=READY pgs=14744 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0075c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:09.955 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268d0 secure :-1 s=READY pgs=14744 cs=0 l=1 rev1=1 rx=0x7f21e00394d0 tx=0x7f21e0075c20).stop 2021-03-09T10:50:09.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:09.956 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268d0 unknown :-1 s=CLOSED pgs=14744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:09.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:09.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:09.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:09.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026260 con 0x7f21c400a6b0 2021-03-09T10:50:09.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057660 con 0x7f21c40037a0 2021-03-09T10:50:09.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021ae0 con 0x7f21c4004620 2021-03-09T10:50:09.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:09.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:09.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:09.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:09.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.949+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:09.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:09.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:09.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:09.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:09.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc092330 con 0x7f21c4004620 2021-03-09T10:50:09.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0942e0 con 0x7f21c4004620 2021-03-09T10:50:09.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.953+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14756 cs=0 l=1 rev1=1 rx=0x7f21dc083660 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:09.962 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:09.957+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50845 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:09.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.957+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:50:09.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0834c0 con 0x7f21c4004620 2021-03-09T10:50:09.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.965+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4004620 2021-03-09T10:50:09.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.965+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:09.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:09.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:09.965+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:14.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:14.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:14.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14756 cs=0 l=1 rev1=1 rx=0x7f21dc083660 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:14.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14756 cs=0 l=1 rev1=1 rx=0x7f21dc083660 tx=0x7f21dc009290).stop 2021-03-09T10:50:14.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:14.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:14.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:14.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:14.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:14.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026260 con 0x7f21c400a6b0 2021-03-09T10:50:14.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057660 con 0x7f21c40037a0 2021-03-09T10:50:14.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026da0 con 0x7f21cc0138f0 2021-03-09T10:50:14.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:14.965 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:14.966 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:14.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:14.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:14.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:14.967 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:14.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:14.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:14.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021ae0 con 0x7f21cc0138f0 2021-03-09T10:50:14.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.961+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017470 con 0x7f21cc0138f0 2021-03-09T10:50:14.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.965+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14771 cs=0 l=1 rev1=1 rx=0x7f21e00903f0 tx=0x7f21e0007de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:14.971 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:14.969+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50882 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:14.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000b3e0 con 0x7f21cc0138f0 2021-03-09T10:50:14.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0076980 con 0x7f21cc0138f0 2021-03-09T10:50:14.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.973+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:50:14.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.973+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:14.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.977+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:14.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:14.977+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:19.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:19.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:19.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14771 cs=0 l=1 rev1=1 rx=0x7f21e00903f0 tx=0x7f21e0007de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:19.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14771 cs=0 l=1 rev1=1 rx=0x7f21e00903f0 tx=0x7f21e0007de0).stop 2021-03-09T10:50:19.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:19.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=CLOSED pgs=14771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:19.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:19.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:19.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:19.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026260 con 0x7f21c400a6b0 2021-03-09T10:50:19.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026da0 con 0x7f21c40037a0 2021-03-09T10:50:19.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057660 con 0x7f21c4004620 2021-03-09T10:50:19.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:19.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:19.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:19.976 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:19.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.969+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:19.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:19.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:19.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:19.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:19.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0942e0 con 0x7f21c4004620 2021-03-09T10:50:19.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:50:19.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.973+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14778 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e003e5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:19.981 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:19.977+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50921 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:19.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00767a0 con 0x7f21c4004620 2021-03-09T10:50:19.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0076980 con 0x7f21c4004620 2021-03-09T10:50:19.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.981+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21c4004620 2021-03-09T10:50:19.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.981+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:19.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:19.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:19.981+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:24.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:24.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:24.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14778 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e003e5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:24.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14778 cs=0 l=1 rev1=1 rx=0x7f21e00a61d0 tx=0x7f21e003e5b0).stop 2021-03-09T10:50:24.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:24.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:24.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:24.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:24.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:24.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e770 con 0x7f21c400a6b0 2021-03-09T10:50:24.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a8d0 con 0x7f21c40037a0 2021-03-09T10:50:24.985 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026260 con 0x7f21cc0138f0 2021-03-09T10:50:24.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:24.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:24.986 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:24.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:24.987 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.977+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:24.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:24.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:24.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:24.988 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:24.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021ae0 con 0x7f21cc0138f0 2021-03-09T10:50:24.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0942e0 con 0x7f21cc0138f0 2021-03-09T10:50:24.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.981+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14787 cs=0 l=1 rev1=1 rx=0x7f21e00a62b0 tx=0x7f21e003e430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:24.990 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:24.985+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50958 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:24.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0090080 con 0x7f21cc0138f0 2021-03-09T10:50:24.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.985+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:50:24.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00ae180 con 0x7f21cc0138f0 2021-03-09T10:50:24.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.989+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:24.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.993+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:24.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:24.993+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:29.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:29.070 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:29.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.985+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:29.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.985+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:29.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.985+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14787 cs=0 l=1 rev1=1 rx=0x7f21e00a62b0 tx=0x7f21e003e430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:29.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.985+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14787 cs=0 l=1 rev1=1 rx=0x7f21e00a62b0 tx=0x7f21e003e430).stop 2021-03-09T10:50:29.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:29.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=CLOSED pgs=14787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:29.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:29.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:29.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:29.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e770 con 0x7f21c400a6b0 2021-03-09T10:50:29.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026da0 con 0x7f21c40037a0 2021-03-09T10:50:29.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026230 con 0x7f21c4004620 2021-03-09T10:50:29.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:29.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:29.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:29.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:29.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:29.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:29.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:29.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:29.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:29.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0942e0 con 0x7f21c4004620 2021-03-09T10:50:29.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.989+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021db0 con 0x7f21c4004620 2021-03-09T10:50:29.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.993+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14798 cs=0 l=1 rev1=1 rx=0x7f21cc00aa80 tx=0x7f21e407d1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:30.000 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:29.997+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50996 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:30.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b0120 con 0x7f21c4004620 2021-03-09T10:50:30.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:29.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017120 con 0x7f21c4004620 2021-03-09T10:50:30.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:30.001+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b0120 con 0x7f21c4004620 2021-03-09T10:50:30.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:30.001+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:30.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:30.001+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:30.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:30.001+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:35.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:50:35.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:35.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14798 cs=0 l=1 rev1=1 rx=0x7f21cc00aa80 tx=0x7f21e407d1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:35.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14798 cs=0 l=1 rev1=1 rx=0x7f21cc00aa80 tx=0x7f21e407d1d0).stop 2021-03-09T10:50:35.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:35.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:35.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:35.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:35.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:35.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04e770 con 0x7f21c400a6b0 2021-03-09T10:50:35.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012290 con 0x7f21c40037a0 2021-03-09T10:50:35.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026da0 con 0x7f21cc0138f0 2021-03-09T10:50:35.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:35.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:35.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:35.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:35.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:34.997+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:35.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:35.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:35.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:35.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:35.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021db0 con 0x7f21cc0138f0 2021-03-09T10:50:35.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:50:35.008 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.001+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14808 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003e620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:35.010 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:35.005+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51034 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:35.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0037c70 con 0x7f21cc0138f0 2021-03-09T10:50:35.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:50:35.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.013+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003e1a0 con 0x7f21cc0138f0 2021-03-09T10:50:35.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.013+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:35.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.013+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:35.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:35.013+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:40.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:40.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:40.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14808 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003e620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:40.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14808 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003e620).stop 2021-03-09T10:50:40.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:40.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=CLOSED pgs=14808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:40.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:40.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:40.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:40.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc012290 con 0x7f21c400a6b0 2021-03-09T10:50:40.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.005+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c0d0 con 0x7f21c40037a0 2021-03-09T10:50:40.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c210 con 0x7f21c4004620 2021-03-09T10:50:40.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:40.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:40.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:40.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:40.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:40.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:40.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:40.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:40.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:40.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ab40 con 0x7f21c4004620 2021-03-09T10:50:40.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021db0 con 0x7f21c4004620 2021-03-09T10:50:40.017 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.009+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14819 cs=0 l=1 rev1=1 rx=0x7f21e000e040 tx=0x7f21e0013880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:40.018 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:40.013+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51072 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:40.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0029110 con 0x7f21c4004620 2021-03-09T10:50:40.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0007de0 con 0x7f21c4004620 2021-03-09T10:50:40.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e002a100 con 0x7f21c4004620 2021-03-09T10:50:40.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.025+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:40.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.025+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:40.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:40.025+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:45.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:45.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:45.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14819 cs=0 l=1 rev1=1 rx=0x7f21e000e040 tx=0x7f21e0013880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:45.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14819 cs=0 l=1 rev1=1 rx=0x7f21e000e040 tx=0x7f21e0013880).stop 2021-03-09T10:50:45.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:45.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:45.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:45.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:45.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:45.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c400a6b0 2021-03-09T10:50:45.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ae70 con 0x7f21c40037a0 2021-03-09T10:50:45.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d680 con 0x7f21cc0138f0 2021-03-09T10:50:45.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:45.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:45.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:45.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:45.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d7f0 con 0x7f21cc0138f0 2021-03-09T10:50:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0942e0 con 0x7f21cc0138f0 2021-03-09T10:50:45.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.017+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14830 cs=0 l=1 rev1=1 rx=0x7f21e407f1b0 tx=0x7f21e4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:45.028 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:45.025+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51110 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:45.028 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:50:45.029 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4016d00 con 0x7f21cc0138f0 2021-03-09T10:50:45.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.029+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:50:45.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.029+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:45.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.029+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:45.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:45.029+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:50.030 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14830 cs=0 l=1 rev1=1 rx=0x7f21e407f1b0 tx=0x7f21e4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14830 cs=0 l=1 rev1=1 rx=0x7f21e407f1b0 tx=0x7f21e4005110).stop 2021-03-09T10:50:50.031 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:50.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=CLOSED pgs=14830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:50.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:50.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:50.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d680 con 0x7f21c400a6b0 2021-03-09T10:50:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c40037a0 2021-03-09T10:50:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0942e0 con 0x7f21c4004620 2021-03-09T10:50:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:50.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:50.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.025+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:50.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:50.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:50.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:50.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:50.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ae70 con 0x7f21c4004620 2021-03-09T10:50:50.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ab40 con 0x7f21c4004620 2021-03-09T10:50:50.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.029+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14842 cs=0 l=1 rev1=1 rx=0x7f21e002e030 tx=0x7f21e0014f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:50.039 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:50.037+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51148 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:50.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0090080 con 0x7f21c4004620 2021-03-09T10:50:50.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e009ae10 con 0x7f21c4004620 2021-03-09T10:50:50.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21c4004620 2021-03-09T10:50:50.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.037+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:50.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.041+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:50.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:50.041+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:55.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:50:55.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:55.040 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14842 cs=0 l=1 rev1=1 rx=0x7f21e002e030 tx=0x7f21e0014f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:55.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14842 cs=0 l=1 rev1=1 rx=0x7f21e002e030 tx=0x7f21e0014f50).stop 2021-03-09T10:50:55.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:55.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:55.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:55.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:55.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:55.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d680 con 0x7f21c400a6b0 2021-03-09T10:50:55.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c40037a0 2021-03-09T10:50:55.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0942e0 con 0x7f21cc0138f0 2021-03-09T10:50:55.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:50:55.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:55.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:50:55.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:55.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:55.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:55.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:55.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:55.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:55.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.037+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ab40 con 0x7f21cc0138f0 2021-03-09T10:50:55.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.041+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc092330 con 0x7f21cc0138f0 2021-03-09T10:50:55.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.041+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14854 cs=0 l=1 rev1=1 rx=0x7f21cc0896f0 tx=0x7f21dc077de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:55.049 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:50:55.045+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51186 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:50:55.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.049+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:50:55.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21cc0138f0 2021-03-09T10:50:55.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.053+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0138f0 2021-03-09T10:50:55.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.053+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:50:55.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.057+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:50:55.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:50:55.057+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:50:59.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:50:59.132 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:00.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:00.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:00.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14854 cs=0 l=1 rev1=1 rx=0x7f21cc0896f0 tx=0x7f21dc077de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:00.051 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14854 cs=0 l=1 rev1=1 rx=0x7f21cc0896f0 tx=0x7f21dc077de0).stop 2021-03-09T10:51:00.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:00.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=CLOSED pgs=14854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:00.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:00.052 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:00.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:00.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc095680 con 0x7f21c400a6b0 2021-03-09T10:51:00.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042ba0 con 0x7f21c40037a0 2021-03-09T10:51:00.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c4004620 2021-03-09T10:51:00.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:00.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:00.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.044+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:00.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:00.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:00.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:00.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:00.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:00.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:00.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0942e0 con 0x7f21c4004620 2021-03-09T10:51:00.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ab40 con 0x7f21c4004620 2021-03-09T10:51:00.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.048+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14863 cs=0 l=1 rev1=1 rx=0x7f21cc00aa80 tx=0x7f21dc031de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:00.057 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:00.052+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51224 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:00.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.060+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083150 con 0x7f21c4004620 2021-03-09T10:51:00.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.060+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0832b0 con 0x7f21c4004620 2021-03-09T10:51:00.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.060+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc087180 con 0x7f21c4004620 2021-03-09T10:51:00.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.060+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:00.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.060+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:00.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:00.060+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:05.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:05.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:05.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14863 cs=0 l=1 rev1=1 rx=0x7f21cc00aa80 tx=0x7f21dc031de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:05.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14863 cs=0 l=1 rev1=1 rx=0x7f21cc00aa80 tx=0x7f21dc031de0).stop 2021-03-09T10:51:05.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:05.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:05.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:05.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:05.061 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:05.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c400a6b0 2021-03-09T10:51:05.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042220 con 0x7f21c40037a0 2021-03-09T10:51:05.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.052+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042360 con 0x7f21cc0138f0 2021-03-09T10:51:05.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:05.062 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:05.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:05.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:05.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:05.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:05.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:05.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:05.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:05.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ae70 con 0x7f21cc0138f0 2021-03-09T10:51:05.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d7f0 con 0x7f21cc0138f0 2021-03-09T10:51:05.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.056+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14872 cs=0 l=1 rev1=1 rx=0x7f21e407d1a0 tx=0x7f21e407f200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:05.067 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:05.064+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51262 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:05.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.064+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21cc0138f0 2021-03-09T10:51:05.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.064+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:51:05.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.068+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21cc0138f0 2021-03-09T10:51:05.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.068+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:05.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.068+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:05.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:05.068+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:10.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:10.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:10.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14872 cs=0 l=1 rev1=1 rx=0x7f21e407d1a0 tx=0x7f21e407f200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:10.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 secure :-1 s=READY pgs=14872 cs=0 l=1 rev1=1 rx=0x7f21e407d1a0 tx=0x7f21e407f200).stop 2021-03-09T10:51:10.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0268b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:10.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0268b0 unknown :-1 s=CLOSED pgs=14872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:10.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:10.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:10.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:10.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c400a6b0 2021-03-09T10:51:10.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d990 con 0x7f21c40037a0 2021-03-09T10:51:10.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a8d0 con 0x7f21c4004620 2021-03-09T10:51:10.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:10.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:10.072 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:10.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:10.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.064+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:10.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:10.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:10.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:10.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:10.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00aa40 con 0x7f21c4004620 2021-03-09T10:51:10.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0942e0 con 0x7f21c4004620 2021-03-09T10:51:10.075 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.068+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14887 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0abf50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:10.078 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:10.072+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:10.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.076+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:51:10.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.080+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:51:10.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.080+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:51:10.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.080+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:10.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.080+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:10.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:10.080+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:15.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.072+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:51:15.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.072+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:15.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.072+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14887 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0abf50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:15.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.072+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14887 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc0abf50).stop 2021-03-09T10:51:15.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:15.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:15.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:15.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:15.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:15.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04b660 con 0x7f21c400a6b0 2021-03-09T10:51:15.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0120d0 con 0x7f21c40037a0 2021-03-09T10:51:15.082 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a8d0 con 0x7f21cc0138f0 2021-03-09T10:51:15.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:15.083 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:15.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:15.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:15.084 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:15.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:15.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:15.085 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:15.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:15.086 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021db0 con 0x7f21cc0138f0 2021-03-09T10:51:15.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ae70 con 0x7f21cc0138f0 2021-03-09T10:51:15.087 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.076+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041ee0 secure :-1 s=READY pgs=14895 cs=0 l=1 rev1=1 rx=0x7f21e404f700 tx=0x7f21e404f540).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:15.088 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:15.084+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51338 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:15.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.084+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:51:15.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b1120 con 0x7f21cc0138f0 2021-03-09T10:51:15.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:51:15.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.088+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:15.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.088+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:15.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:15.088+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:20.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:20.090 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:20.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041ee0 secure :-1 s=READY pgs=14895 cs=0 l=1 rev1=1 rx=0x7f21e404f700 tx=0x7f21e404f540).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:20.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041ee0 secure :-1 s=READY pgs=14895 cs=0 l=1 rev1=1 rx=0x7f21e404f700 tx=0x7f21e404f540).stop 2021-03-09T10:51:20.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:20.091 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041ee0 unknown :-1 s=CLOSED pgs=14895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:20.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:20.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:20.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:20.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0120d0 con 0x7f21c400a6b0 2021-03-09T10:51:20.092 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0355b0 con 0x7f21c40037a0 2021-03-09T10:51:20.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a8d0 con 0x7f21c4004620 2021-03-09T10:51:20.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:20.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:20.093 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:20.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:20.094 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:20.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:20.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:20.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:20.095 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:20.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d990 con 0x7f21c4004620 2021-03-09T10:51:20.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:51:20.096 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.088+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14904 cs=0 l=1 rev1=1 rx=0x7f21e00af000 tx=0x7f21e002b250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:20.098 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:20.092+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51376 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:20.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4004620 2021-03-09T10:51:20.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4004620 2021-03-09T10:51:20.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.100+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21c4004620 2021-03-09T10:51:20.106 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.100+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:20.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:20.107 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:20.104+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:25.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:25.099 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:25.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14904 cs=0 l=1 rev1=1 rx=0x7f21e00af000 tx=0x7f21e002b250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:25.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14904 cs=0 l=1 rev1=1 rx=0x7f21e00af000 tx=0x7f21e002b250).stop 2021-03-09T10:51:25.100 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:25.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:25.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:25.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.092+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:25.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:25.101 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:25.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:25.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0120d0 con 0x7f21c400a6b0 2021-03-09T10:51:25.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0355b0 con 0x7f21c40037a0 2021-03-09T10:51:25.102 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a730 con 0x7f21cc0138f0 2021-03-09T10:51:25.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:25.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:25.103 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:25.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:25.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:25.104 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:25.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:25.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0942e0 con 0x7f21cc0138f0 2021-03-09T10:51:25.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021db0 con 0x7f21cc0138f0 2021-03-09T10:51:25.105 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.096+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041e30 secure :-1 s=READY pgs=14916 cs=0 l=1 rev1=1 rx=0x7f21cc00f310 tx=0x7f21e40b4670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:25.109 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:25.104+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51414 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:25.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b1330 con 0x7f21cc0138f0 2021-03-09T10:51:25.109 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.104+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f380 con 0x7f21cc0138f0 2021-03-09T10:51:25.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21cc0138f0 2021-03-09T10:51:25.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.108+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:25.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:25.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:25.108+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:29.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:29.188 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:30.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:30.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:30.111 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041e30 secure :-1 s=READY pgs=14916 cs=0 l=1 rev1=1 rx=0x7f21cc00f310 tx=0x7f21e40b4670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:30.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041e30 secure :-1 s=READY pgs=14916 cs=0 l=1 rev1=1 rx=0x7f21cc00f310 tx=0x7f21e40b4670).stop 2021-03-09T10:51:30.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:30.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041e30 unknown :-1 s=CLOSED pgs=14916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:30.112 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:30.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:30.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:30.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0120d0 con 0x7f21c400a6b0 2021-03-09T10:51:30.113 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a380 con 0x7f21c40037a0 2021-03-09T10:51:30.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021db0 con 0x7f21c4004620 2021-03-09T10:51:30.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:30.114 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:30.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:30.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:30.115 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:30.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:30.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:30.116 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:30.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:30.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc028160 con 0x7f21c4004620 2021-03-09T10:51:30.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00aa40 con 0x7f21c4004620 2021-03-09T10:51:30.117 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.108+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14928 cs=0 l=1 rev1=1 rx=0x7f21dc0b5060 tx=0x7f21dc032220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:30.118 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:30.116+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51452 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:30.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4004620 2021-03-09T10:51:30.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4004620 2021-03-09T10:51:30.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.120+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:51:30.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.120+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:30.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.120+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:30.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:30.120+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:35.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:35.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:35.120 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14928 cs=0 l=1 rev1=1 rx=0x7f21dc0b5060 tx=0x7f21dc032220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:35.121 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14928 cs=0 l=1 rev1=1 rx=0x7f21dc0b5060 tx=0x7f21dc032220).stop 2021-03-09T10:51:35.122 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:35.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:35.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:35.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:35.123 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:35.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0120d0 con 0x7f21c400a6b0 2021-03-09T10:51:35.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0282f0 con 0x7f21c40037a0 2021-03-09T10:51:35.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0324e0 con 0x7f21cc0138f0 2021-03-09T10:51:35.124 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:35.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:35.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:35.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:35.125 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:35.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:35.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:35.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:35.126 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:35.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021db0 con 0x7f21cc0138f0 2021-03-09T10:51:35.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d990 con 0x7f21cc0138f0 2021-03-09T10:51:35.127 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.116+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14937 cs=0 l=1 rev1=1 rx=0x7f21e0007d70 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:35.129 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:35.124+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51490 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:35.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:51:35.130 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.124+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21cc0138f0 2021-03-09T10:51:35.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:51:35.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.128+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:35.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.128+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:35.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:35.128+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:40.131 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:40.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:40.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14937 cs=0 l=1 rev1=1 rx=0x7f21e0007d70 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:40.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14937 cs=0 l=1 rev1=1 rx=0x7f21e0007d70 tx=0x7f21e0003820).stop 2021-03-09T10:51:40.132 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:40.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=14937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:40.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:40.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:40.133 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:40.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0282f0 con 0x7f21c400a6b0 2021-03-09T10:51:40.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023610 con 0x7f21c40037a0 2021-03-09T10:51:40.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c4004620 2021-03-09T10:51:40.134 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:40.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:40.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:40.135 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:40.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:40.136 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:40.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:40.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:40.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:40.137 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d990 con 0x7f21c4004620 2021-03-09T10:51:40.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc028160 con 0x7f21c4004620 2021-03-09T10:51:40.138 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.128+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14948 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc029190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:40.139 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:40.132+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51528 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:40.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4004620 2021-03-09T10:51:40.139 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21c4004620 2021-03-09T10:51:40.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:51:40.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.136+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:40.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.136+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:40.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:40.136+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:45.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:45.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:45.140 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14948 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc029190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:45.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14948 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc029190).stop 2021-03-09T10:51:45.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:45.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:45.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:45.141 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:45.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:45.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:51:45.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a380 con 0x7f21c40037a0 2021-03-09T10:51:45.142 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a4c0 con 0x7f21cc0138f0 2021-03-09T10:51:45.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:45.143 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:45.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:45.144 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:45.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:45.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:45.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:45.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:45.145 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.136+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:45.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.140+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042d90 con 0x7f21cc0138f0 2021-03-09T10:51:45.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.140+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021db0 con 0x7f21cc0138f0 2021-03-09T10:51:45.146 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.140+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14962 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007d5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:45.149 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:45.144+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:45.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.144+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0013070 con 0x7f21cc0138f0 2021-03-09T10:51:45.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.152+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0013070 con 0x7f21cc0138f0 2021-03-09T10:51:45.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.152+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:51:45.159 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.152+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:45.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.152+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:45.160 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:45.152+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:50.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:50.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:50.151 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14962 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007d5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:50.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14962 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007d5b0).stop 2021-03-09T10:51:50.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:50.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=14962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:50.152 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:51:50.153 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0531b0 con 0x7f21c40037a0 2021-03-09T10:51:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:51:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:50.154 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.144+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:50.155 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:50.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:50.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:50.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:50.156 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:50.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03a380 con 0x7f21c4004620 2021-03-09T10:51:50.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d990 con 0x7f21c4004620 2021-03-09T10:51:50.157 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.148+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14969 cs=0 l=1 rev1=1 rx=0x7f21dc033250 tx=0x7f21dc077e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:50.160 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:50.156+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51604 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:50.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.156+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0330b0 con 0x7f21c4004620 2021-03-09T10:51:50.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032ee0 con 0x7f21c4004620 2021-03-09T10:51:50.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.164+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:51:50.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.164+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:50.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.164+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:50.170 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:50.164+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:55.162 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:51:55.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:55.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14969 cs=0 l=1 rev1=1 rx=0x7f21dc033250 tx=0x7f21dc077e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:55.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14969 cs=0 l=1 rev1=1 rx=0x7f21dc033250 tx=0x7f21dc077e60).stop 2021-03-09T10:51:55.163 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:55.164 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:51:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0531b0 con 0x7f21c40037a0 2021-03-09T10:51:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046c00 con 0x7f21cc0138f0 2021-03-09T10:51:55.165 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:51:55.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:55.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:51:55.166 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:55.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.155+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:55.167 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:55.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:55.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:55.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:55.168 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011800 con 0x7f21cc0138f0 2021-03-09T10:51:55.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0942e0 con 0x7f21cc0138f0 2021-03-09T10:51:55.169 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.159+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14981 cs=0 l=1 rev1=1 rx=0x7f21e40b4520 tx=0x7f21e4053fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:55.171 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:51:55.167+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51642 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:51:55.172 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.167+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc0138f0 2021-03-09T10:51:55.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21cc0138f0 2021-03-09T10:51:55.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e404f130 con 0x7f21cc0138f0 2021-03-09T10:51:55.178 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.171+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:51:55.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.175+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:51:55.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:51:55.175+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:51:59.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:51:59.245 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:00.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:00.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:00.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14981 cs=0 l=1 rev1=1 rx=0x7f21e40b4520 tx=0x7f21e4053fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:00.173 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=14981 cs=0 l=1 rev1=1 rx=0x7f21e40b4520 tx=0x7f21e4053fb0).stop 2021-03-09T10:52:00.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:00.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=14981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:00.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:00.174 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:00.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:00.175 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:52:00.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c40037a0 2021-03-09T10:52:00.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d7f0 con 0x7f21c4004620 2021-03-09T10:52:00.176 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:00.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:00.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:00.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:00.177 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:00.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:00.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:00.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:00.179 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:00.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08d930 con 0x7f21c4004620 2021-03-09T10:52:00.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011800 con 0x7f21c4004620 2021-03-09T10:52:00.180 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.171+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14990 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4015ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:00.181 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:00.175+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:00.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.175+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21c4004620 2021-03-09T10:52:00.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.175+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4008ac0 con 0x7f21c4004620 2021-03-09T10:52:00.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.179+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008ac0 con 0x7f21c4004620 2021-03-09T10:52:00.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.179+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:00.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.179+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:00.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:00.179+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:05.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:52:05.181 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:05.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14990 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4015ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:05.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=14990 cs=0 l=1 rev1=1 rx=0x7f21e4028510 tx=0x7f21e4015ea0).stop 2021-03-09T10:52:05.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:05.182 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=14990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:05.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:05.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:05.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:05.183 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:52:05.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d7c0 con 0x7f21c40037a0 2021-03-09T10:52:05.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:05.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056f40 con 0x7f21cc0138f0 2021-03-09T10:52:05.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:05.184 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:05.185 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:05.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.179+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:05.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:05.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:05.186 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:05.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:05.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057080 con 0x7f21cc0138f0 2021-03-09T10:52:05.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03a380 con 0x7f21cc0138f0 2021-03-09T10:52:05.187 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.183+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15002 cs=0 l=1 rev1=1 rx=0x7f21dc007da0 tx=0x7f21dc0205a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:05.189 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:05.187+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51718 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:05.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.187+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0138f0 2021-03-09T10:52:05.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.191+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0138f0 2021-03-09T10:52:05.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.195+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:52:05.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.195+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:05.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.195+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:05.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:05.195+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:10.190 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:10.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:10.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15002 cs=0 l=1 rev1=1 rx=0x7f21dc007da0 tx=0x7f21dc0205a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:10.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15002 cs=0 l=1 rev1=1 rx=0x7f21dc007da0 tx=0x7f21dc0205a0).stop 2021-03-09T10:52:10.191 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:10.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=15002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:10.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:10.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:10.192 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:10.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:52:10.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:52:10.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:52:10.193 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:10.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:10.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:10.194 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:10.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.187+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:10.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:10.195 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:10.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:10.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:10.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0170e0 con 0x7f21c4004620 2021-03-09T10:52:10.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08d930 con 0x7f21c4004620 2021-03-09T10:52:10.196 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.191+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=15013 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4091210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:10.199 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:10.199+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:10.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4005340 con 0x7f21c4004620 2021-03-09T10:52:10.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.203+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4005340 con 0x7f21c4004620 2021-03-09T10:52:10.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028240 con 0x7f21c4004620 2021-03-09T10:52:10.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.207+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:10.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.207+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:10.208 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:10.207+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:15.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:15.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:15.200 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=15013 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4091210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:15.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=15013 cs=0 l=1 rev1=1 rx=0x7f21e4005500 tx=0x7f21e4091210).stop 2021-03-09T10:52:15.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:15.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=15013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:15.201 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:15.202 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:15.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:15.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c400a6b0 2021-03-09T10:52:15.203 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c40037a0 2021-03-09T10:52:15.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017270 con 0x7f21cc0138f0 2021-03-09T10:52:15.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:15.204 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:15.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:15.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:15.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:15.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:15.205 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:15.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:15.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:15.206 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056d50 con 0x7f21cc0138f0 2021-03-09T10:52:15.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.199+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0170e0 con 0x7f21cc0138f0 2021-03-09T10:52:15.207 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.203+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15022 cs=0 l=1 rev1=1 rx=0x7f21cc01dbc0 tx=0x7f21e402d8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:15.208 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:15.207+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51794 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:15.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4028240 con 0x7f21cc0138f0 2021-03-09T10:52:15.209 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4059000 con 0x7f21cc0138f0 2021-03-09T10:52:15.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.211+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4028240 con 0x7f21cc0138f0 2021-03-09T10:52:15.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.211+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:15.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.211+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:15.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:15.211+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:20.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:20.210 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:20.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15022 cs=0 l=1 rev1=1 rx=0x7f21cc01dbc0 tx=0x7f21e402d8d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:20.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15022 cs=0 l=1 rev1=1 rx=0x7f21cc01dbc0 tx=0x7f21e402d8d0).stop 2021-03-09T10:52:20.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:20.211 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=15022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:20.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:20.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:20.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:20.212 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:52:20.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c40037a0 2021-03-09T10:52:20.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021950 con 0x7f21c4004620 2021-03-09T10:52:20.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:20.213 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:20.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:20.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:20.214 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.207+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:20.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:20.215 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:20.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:20.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:20.216 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0418c0 con 0x7f21c4004620 2021-03-09T10:52:20.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057080 con 0x7f21c4004620 2021-03-09T10:52:20.217 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.211+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=15033 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:20.219 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51832 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:20.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc031de0 con 0x7f21c4004620 2021-03-09T10:52:20.220 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc032010 con 0x7f21c4004620 2021-03-09T10:52:20.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:52:20.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:20.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:20.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:20.219+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:25.221 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:25.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:25.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=15033 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:25.222 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 secure :-1 s=READY pgs=15033 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc009290).stop 2021-03-09T10:52:25.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc021460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:25.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc021460 unknown :-1 s=CLOSED pgs=15033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:25.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:25.223 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:25.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:25.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017ee0 con 0x7f21c400a6b0 2021-03-09T10:52:25.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c40037a0 2021-03-09T10:52:25.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ad60 con 0x7f21cc0138f0 2021-03-09T10:52:25.224 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:25.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:25.225 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:25.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:25.226 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.219+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:25.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:25.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:25.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:25.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:25.227 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056ee0 con 0x7f21cc0138f0 2021-03-09T10:52:25.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0418c0 con 0x7f21cc0138f0 2021-03-09T10:52:25.228 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.223+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15046 cs=0 l=1 rev1=1 rx=0x7f21dc0abeb0 tx=0x7f21dc0b6800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:25.230 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:25.227+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51870 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:25.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:52:25.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc015a20 con 0x7f21cc0138f0 2021-03-09T10:52:25.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.235+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0b61d0 con 0x7f21cc0138f0 2021-03-09T10:52:25.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.235+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:25.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.235+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:25.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:25.235+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:29.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:29.291 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:30.231 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:30.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:30.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15046 cs=0 l=1 rev1=1 rx=0x7f21dc0abeb0 tx=0x7f21dc0b6800).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:30.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15046 cs=0 l=1 rev1=1 rx=0x7f21dc0abeb0 tx=0x7f21dc0b6800).stop 2021-03-09T10:52:30.232 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:30.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=15046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:30.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:30.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:30.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:30.233 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c400a6b0 2021-03-09T10:52:30.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0215a0 con 0x7f21c40037a0 2021-03-09T10:52:30.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0216e0 con 0x7f21c4004620 2021-03-09T10:52:30.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:30.234 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:30.235 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:30.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:30.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:30.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:30.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:30.236 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:30.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:30.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021820 con 0x7f21c4004620 2021-03-09T10:52:30.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042d90 con 0x7f21c4004620 2021-03-09T10:52:30.237 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.231+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0416d0 secure :-1 s=READY pgs=15054 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e00291f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:30.241 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:30.239+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51908 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:30.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:52:30.242 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21c4004620 2021-03-09T10:52:30.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.243+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:52:30.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.243+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:30.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.243+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:30.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:30.243+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:35.243 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:35.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:35.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0416d0 secure :-1 s=READY pgs=15054 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e00291f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:35.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0416d0 secure :-1 s=READY pgs=15054 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e00291f0).stop 2021-03-09T10:52:35.244 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0416d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:35.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0416d0 unknown :-1 s=CLOSED pgs=15054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:35.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:35.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:35.245 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:35.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21c400a6b0 2021-03-09T10:52:35.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:52:35.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0215a0 con 0x7f21cc0138f0 2021-03-09T10:52:35.246 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:35.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:35.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:35.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.239+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:35.247 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:35.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:35.248 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:35.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:35.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:35.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0115b0 con 0x7f21cc0138f0 2021-03-09T10:52:35.249 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056ee0 con 0x7f21cc0138f0 2021-03-09T10:52:35.250 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.243+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15065 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc019f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:35.253 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:35.251+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51945 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:35.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc0138f0 2021-03-09T10:52:35.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0bada0 con 0x7f21cc0138f0 2021-03-09T10:52:35.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.259+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:52:35.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.259+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:35.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.259+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:35.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:35.259+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:40.254 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:40.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:40.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15065 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc019f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:40.255 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15065 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc019f50).stop 2021-03-09T10:52:40.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:40.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=15065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:40.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:40.256 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:40.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:40.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc041a20 con 0x7f21c400a6b0 2021-03-09T10:52:40.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:52:40.257 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0215a0 con 0x7f21c4004620 2021-03-09T10:52:40.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:40.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:40.258 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:40.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:40.259 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.251+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:40.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:40.260 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:40.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:40.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:40.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011740 con 0x7f21c4004620 2021-03-09T10:52:40.261 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056d50 con 0x7f21c4004620 2021-03-09T10:52:40.262 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.255+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15076 cs=0 l=1 rev1=1 rx=0x7f21e407f330 tx=0x7f21e407d1a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:40.262 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:40.259+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51984 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:40.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4004620 2021-03-09T10:52:40.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009390 con 0x7f21c4004620 2021-03-09T10:52:40.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.263+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:52:40.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.263+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:40.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.267+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:40.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:40.267+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:45.263 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:45.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:45.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15076 cs=0 l=1 rev1=1 rx=0x7f21e407f330 tx=0x7f21e407d1a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:45.264 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15076 cs=0 l=1 rev1=1 rx=0x7f21e407f330 tx=0x7f21e407d1a0).stop 2021-03-09T10:52:45.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:45.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:45.265 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:45.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:45.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:45.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a140 con 0x7f21c400a6b0 2021-03-09T10:52:45.266 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c40037a0 2021-03-09T10:52:45.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00ac20 con 0x7f21cc0138f0 2021-03-09T10:52:45.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:45.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:45.267 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:45.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:45.268 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:45.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:45.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:45.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:45.269 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:45.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0922e0 con 0x7f21cc0138f0 2021-03-09T10:52:45.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021820 con 0x7f21cc0138f0 2021-03-09T10:52:45.270 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.262+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15086 cs=0 l=1 rev1=1 rx=0x7f21e003ddd0 tx=0x7f21e000b450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:45.273 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:45.270+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52021 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:45.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21cc0138f0 2021-03-09T10:52:45.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0059ab0 con 0x7f21cc0138f0 2021-03-09T10:52:45.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.274+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0059d80 con 0x7f21cc0138f0 2021-03-09T10:52:45.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.274+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:45.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.274+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:45.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:45.274+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:50.274 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:50.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:50.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15086 cs=0 l=1 rev1=1 rx=0x7f21e003ddd0 tx=0x7f21e000b450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:50.275 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 secure :-1 s=READY pgs=15086 cs=0 l=1 rev1=1 rx=0x7f21e003ddd0 tx=0x7f21e000b450).stop 2021-03-09T10:52:50.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:50.276 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a730 unknown :-1 s=CLOSED pgs=15086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:50.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:50.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:50.277 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:50.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b000 con 0x7f21c400a6b0 2021-03-09T10:52:50.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c40037a0 2021-03-09T10:52:50.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021820 con 0x7f21c4004620 2021-03-09T10:52:50.278 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.270+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:50.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:50.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:50.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:50.279 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:50.280 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:50.281 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21c4004620 2021-03-09T10:52:50.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.274+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011740 con 0x7f21c4004620 2021-03-09T10:52:50.282 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.278+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15098 cs=0 l=1 rev1=1 rx=0x7f21e402d8a0 tx=0x7f21e40b4350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:50.283 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:50.282+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52060 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:50.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21c4004620 2021-03-09T10:52:50.284 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21c4004620 2021-03-09T10:52:50.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.286+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:52:50.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.286+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:50.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.286+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:50.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:50.286+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:55.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:55.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:55.285 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15098 cs=0 l=1 rev1=1 rx=0x7f21e402d8a0 tx=0x7f21e40b4350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15098 cs=0 l=1 rev1=1 rx=0x7f21e402d8a0 tx=0x7f21e40b4350).stop 2021-03-09T10:52:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:55.286 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021820 con 0x7f21c400a6b0 2021-03-09T10:52:55.287 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b000 con 0x7f21c40037a0 2021-03-09T10:52:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21cc0138f0 2021-03-09T10:52:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:52:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:52:55.288 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:55.289 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:55.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:55.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:55.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:55.290 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21cc0138f0 2021-03-09T10:52:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.282+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011740 con 0x7f21cc0138f0 2021-03-09T10:52:55.291 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.286+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a6e0 secure :-1 s=READY pgs=15108 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e4003bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:55.293 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:52:55.290+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52097 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:52:55.294 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:52:55.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.298+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400fe60 con 0x7f21cc0138f0 2021-03-09T10:52:55.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.298+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0138f0 2021-03-09T10:52:55.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.298+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:52:55.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.298+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:52:55.302 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:52:55.298+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:52:59.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:52:59.339 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:00.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:00.295 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a6e0 secure :-1 s=READY pgs=15108 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e4003bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a6e0 secure :-1 s=READY pgs=15108 cs=0 l=1 rev1=1 rx=0x7f21e407e6a0 tx=0x7f21e4003bf0).stop 2021-03-09T10:53:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:00.296 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a6e0 unknown :-1 s=CLOSED pgs=15108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08df50 con 0x7f21c400a6b0 2021-03-09T10:53:00.297 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021820 con 0x7f21c40037a0 2021-03-09T10:53:00.298 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c4004620 2021-03-09T10:53:00.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:00.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:00.299 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:00.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:00.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:00.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:00.300 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00b1a0 con 0x7f21c4004620 2021-03-09T10:53:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0922e0 con 0x7f21c4004620 2021-03-09T10:53:00.301 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.294+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15118 cs=0 l=1 rev1=1 rx=0x7f21e0091920 tx=0x7f21e002b1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:00.304 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:00.302+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52136 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:00.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0037b60 con 0x7f21c4004620 2021-03-09T10:53:00.305 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0059810 con 0x7f21c4004620 2021-03-09T10:53:00.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:53:00.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.302+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:00.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.306+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:00.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:00.306+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:05.306 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:53:05.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:05.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15118 cs=0 l=1 rev1=1 rx=0x7f21e0091920 tx=0x7f21e002b1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:05.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15118 cs=0 l=1 rev1=1 rx=0x7f21e0091920 tx=0x7f21e002b1d0).stop 2021-03-09T10:53:05.307 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:05.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:05.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:05.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:05.308 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:05.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:05.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:05.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08a110 con 0x7f21c400a6b0 2021-03-09T10:53:05.309 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021820 con 0x7f21c40037a0 2021-03-09T10:53:05.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21cc0138f0 2021-03-09T10:53:05.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:05.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:05.310 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.302+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:05.311 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:05.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:05.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:05.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:05.312 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00ab80 con 0x7f21cc0138f0 2021-03-09T10:53:05.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0115b0 con 0x7f21cc0138f0 2021-03-09T10:53:05.313 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.306+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 secure :-1 s=READY pgs=15129 cs=0 l=1 rev1=1 rx=0x7f21cc021bd0 tx=0x7f21dc015a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:05.315 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:05.314+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52173 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:05.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089e30 con 0x7f21cc0138f0 2021-03-09T10:53:05.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089e30 con 0x7f21cc0138f0 2021-03-09T10:53:05.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.318+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:53:05.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.318+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:05.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.318+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:05.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:05.318+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:10.316 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:10.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:10.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 secure :-1 s=READY pgs=15129 cs=0 l=1 rev1=1 rx=0x7f21cc021bd0 tx=0x7f21dc015a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:10.317 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 secure :-1 s=READY pgs=15129 cs=0 l=1 rev1=1 rx=0x7f21cc021bd0 tx=0x7f21dc015a20).stop 2021-03-09T10:53:10.318 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc00a690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:10.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc00a690 unknown :-1 s=CLOSED pgs=15129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:10.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:10.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:10.319 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:10.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021820 con 0x7f21c400a6b0 2021-03-09T10:53:10.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c40037a0 2021-03-09T10:53:10.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:53:10.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:10.320 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:10.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:10.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:10.321 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.314+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:10.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:10.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:10.322 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:10.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:10.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21c4004620 2021-03-09T10:53:10.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b1a0 con 0x7f21c4004620 2021-03-09T10:53:10.323 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.318+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15141 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e00061e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:10.325 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:10.322+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52212 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:10.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.322+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:53:10.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:53:10.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.330+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006670 con 0x7f21c4004620 2021-03-09T10:53:10.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.330+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:10.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:10.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:10.330+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:15.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:15.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:15.326 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15141 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e00061e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:15.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 secure :-1 s=READY pgs=15141 cs=0 l=1 rev1=1 rx=0x7f21e007e540 tx=0x7f21e00061e0).stop 2021-03-09T10:53:15.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:15.327 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:15.328 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:15.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:15.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:15.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02d130 con 0x7f21c400a6b0 2021-03-09T10:53:15.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c40037a0 2021-03-09T10:53:15.329 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021820 con 0x7f21cc0138f0 2021-03-09T10:53:15.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:15.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:15.330 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:15.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.322+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:15.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:15.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:15.331 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:15.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:15.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:15.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21cc0138f0 2021-03-09T10:53:15.332 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ab80 con 0x7f21cc0138f0 2021-03-09T10:53:15.333 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.326+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 secure :-1 s=READY pgs=15150 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc077e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:15.337 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:15.334+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52249 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:15.337 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc0138f0 2021-03-09T10:53:15.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc018150 con 0x7f21cc0138f0 2021-03-09T10:53:15.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.338+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21cc0138f0 2021-03-09T10:53:15.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.338+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:15.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.338+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:15.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:15.338+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:20.338 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:20.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:20.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 secure :-1 s=READY pgs=15150 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc077e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:20.339 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 secure :-1 s=READY pgs=15150 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc077e50).stop 2021-03-09T10:53:20.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0118d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:20.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 unknown :-1 s=CLOSED pgs=15150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:20.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:20.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:20.340 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:20.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f060 con 0x7f21c400a6b0 2021-03-09T10:53:20.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094590 con 0x7f21c40037a0 2021-03-09T10:53:20.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c4004620 2021-03-09T10:53:20.341 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:20.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:20.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:20.342 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.334+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:20.343 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:20.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:20.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:20.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:20.344 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:20.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021820 con 0x7f21c4004620 2021-03-09T10:53:20.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.338+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00b1a0 con 0x7f21c4004620 2021-03-09T10:53:20.345 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.342+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 secure :-1 s=READY pgs=15163 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0033c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:20.346 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:20.342+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52288 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:20.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0091e90 con 0x7f21c4004620 2021-03-09T10:53:20.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0033e20 con 0x7f21c4004620 2021-03-09T10:53:20.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.350+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:53:20.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.350+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:20.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:20.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:20.350+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:25.347 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:25.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:25.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 secure :-1 s=READY pgs=15163 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0033c50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:25.348 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 secure :-1 s=READY pgs=15163 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0033c50).stop 2021-03-09T10:53:25.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:25.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 unknown :-1 s=CLOSED pgs=15163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:25.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:25.349 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:25.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:25.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f200 con 0x7f21c400a6b0 2021-03-09T10:53:25.350 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc047a50 con 0x7f21c40037a0 2021-03-09T10:53:25.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21cc0138f0 2021-03-09T10:53:25.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:25.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:25.351 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:25.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:25.352 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:25.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:25.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:25.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:25.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:25.353 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021820 con 0x7f21cc0138f0 2021-03-09T10:53:25.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.346+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0138f0 2021-03-09T10:53:25.354 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.350+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 secure :-1 s=READY pgs=15172 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003e260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:25.356 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:25.354+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52325 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:25.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0091e90 con 0x7f21cc0138f0 2021-03-09T10:53:25.357 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0033e20 con 0x7f21cc0138f0 2021-03-09T10:53:25.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.358+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0006920 con 0x7f21cc0138f0 2021-03-09T10:53:25.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.358+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:25.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:25.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:25.358+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:29.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:29.381 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:30.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:30.358 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:30.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 secure :-1 s=READY pgs=15172 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003e260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:30.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 secure :-1 s=READY pgs=15172 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003e260).stop 2021-03-09T10:53:30.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc0118d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:30.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc0118d0 unknown :-1 s=CLOSED pgs=15172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:30.359 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:30.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:30.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:30.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f200 con 0x7f21c400a6b0 2021-03-09T10:53:30.360 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f390 con 0x7f21c40037a0 2021-03-09T10:53:30.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c4004620 2021-03-09T10:53:30.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:30.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:30.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:30.361 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:30.362 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.354+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:30.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:30.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:30.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:30.363 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:30.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc094710 con 0x7f21c4004620 2021-03-09T10:53:30.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011740 con 0x7f21c4004620 2021-03-09T10:53:30.364 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.358+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 secure :-1 s=READY pgs=15182 cs=0 l=1 rev1=1 rx=0x7f21cc02a240 tx=0x7f21e407f170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:30.368 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:30.366+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52364 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:30.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e407f350 con 0x7f21c4004620 2021-03-09T10:53:30.368 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e407f580 con 0x7f21c4004620 2021-03-09T10:53:30.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.370+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4024000 con 0x7f21c4004620 2021-03-09T10:53:30.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.370+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:30.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:30.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:30.370+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:35.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:35.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:35.369 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 secure :-1 s=READY pgs=15182 cs=0 l=1 rev1=1 rx=0x7f21cc02a240 tx=0x7f21e407f170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:35.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 secure :-1 s=READY pgs=15182 cs=0 l=1 rev1=1 rx=0x7f21cc02a240 tx=0x7f21e407f170).stop 2021-03-09T10:53:35.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:35.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011dc0 unknown :-1 s=CLOSED pgs=15182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:35.370 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:35.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:35.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:35.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f390 con 0x7f21c400a6b0 2021-03-09T10:53:35.371 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043790 con 0x7f21c40037a0 2021-03-09T10:53:35.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f200 con 0x7f21cc0138f0 2021-03-09T10:53:35.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:35.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:35.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:35.372 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:35.373 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:35.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:35.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:35.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:35.374 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:35.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc029b00 con 0x7f21cc0138f0 2021-03-09T10:53:35.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.366+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021820 con 0x7f21cc0138f0 2021-03-09T10:53:35.375 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.370+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15193 cs=0 l=1 rev1=1 rx=0x7f21e00144c0 tx=0x7f21e0003820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:35.378 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:35.374+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52401 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:35.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00931d0 con 0x7f21cc0138f0 2021-03-09T10:53:35.378 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.374+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:53:35.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.378+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0138f0 2021-03-09T10:53:35.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.378+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:35.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.378+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:35.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:35.378+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:40.379 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:40.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:40.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15193 cs=0 l=1 rev1=1 rx=0x7f21e00144c0 tx=0x7f21e0003820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:40.380 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15193 cs=0 l=1 rev1=1 rx=0x7f21e00144c0 tx=0x7f21e0003820).stop 2021-03-09T10:53:40.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:40.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:40.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:40.381 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:40.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:40.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043930 con 0x7f21c400a6b0 2021-03-09T10:53:40.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f200 con 0x7f21c40037a0 2021-03-09T10:53:40.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d2b0 con 0x7f21c4004620 2021-03-09T10:53:40.382 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:40.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:40.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:40.383 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:40.384 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:40.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:40.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:40.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:40.385 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:40.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023640 con 0x7f21c4004620 2021-03-09T10:53:40.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.377+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc094710 con 0x7f21c4004620 2021-03-09T10:53:40.386 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.381+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15206 cs=0 l=1 rev1=1 rx=0x7f21e4008000 tx=0x7f21e4021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:40.387 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:40.385+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52440 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:40.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:53:40.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.389+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4004620 2021-03-09T10:53:40.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055180 con 0x7f21c4004620 2021-03-09T10:53:40.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.393+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:40.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.393+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:40.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:40.393+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:45.387 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:45.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:45.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15206 cs=0 l=1 rev1=1 rx=0x7f21e4008000 tx=0x7f21e4021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:45.388 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15206 cs=0 l=1 rev1=1 rx=0x7f21e4008000 tx=0x7f21e4021000).stop 2021-03-09T10:53:45.389 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:45.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:45.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:45.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:45.390 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:45.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d2b0 con 0x7f21c400a6b0 2021-03-09T10:53:45.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043930 con 0x7f21c40037a0 2021-03-09T10:53:45.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00f1b0 con 0x7f21cc0138f0 2021-03-09T10:53:45.391 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:45.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:45.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:45.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:45.392 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:45.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:45.393 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:45.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:45.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:45.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21cc0138f0 2021-03-09T10:53:45.394 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.385+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023640 con 0x7f21cc0138f0 2021-03-09T10:53:45.395 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.389+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15216 cs=0 l=1 rev1=1 rx=0x7f21cc012460 tx=0x7f21e408a530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:45.396 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:45.393+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52477 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:45.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027de0 con 0x7f21cc0138f0 2021-03-09T10:53:45.397 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.393+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408ab30 con 0x7f21cc0138f0 2021-03-09T10:53:45.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408ab30 con 0x7f21cc0138f0 2021-03-09T10:53:45.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.397+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:45.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:45.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:45.397+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:50.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:50.398 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:50.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15216 cs=0 l=1 rev1=1 rx=0x7f21cc012460 tx=0x7f21e408a530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:50.399 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15216 cs=0 l=1 rev1=1 rx=0x7f21cc012460 tx=0x7f21e408a530).stop 2021-03-09T10:53:50.400 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:50.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:50.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:50.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:50.401 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:50.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:50.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d2d0 con 0x7f21c400a6b0 2021-03-09T10:53:50.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:50.402 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d8a0 con 0x7f21c40037a0 2021-03-09T10:53:50.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c810 con 0x7f21c4004620 2021-03-09T10:53:50.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:50.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:50.403 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.397+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:50.404 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:50.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:50.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:50.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:50.405 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00f1b0 con 0x7f21c4004620 2021-03-09T10:53:50.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00ab80 con 0x7f21c4004620 2021-03-09T10:53:50.406 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15232 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0ab0d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:50.406 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:50.401+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52516 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:50.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc081000 con 0x7f21c4004620 2021-03-09T10:53:50.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc081160 con 0x7f21c4004620 2021-03-09T10:53:50.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.409+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4004620 2021-03-09T10:53:50.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.409+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:50.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:50.415 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:50.413+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:55.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:55.407 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:55.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15232 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0ab0d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:55.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15232 cs=0 l=1 rev1=1 rx=0x7f21dc003690 tx=0x7f21dc0ab0d0).stop 2021-03-09T10:53:55.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:55.408 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:55.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:55.409 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:55.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:55.410 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d110 con 0x7f21c400a6b0 2021-03-09T10:53:55.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d250 con 0x7f21c40037a0 2021-03-09T10:53:55.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d6b0 con 0x7f21cc0138f0 2021-03-09T10:53:55.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:53:55.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:55.411 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:55.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:53:55.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:55.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:55.412 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:55.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:55.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:55.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03d7f0 con 0x7f21cc0138f0 2021-03-09T10:53:55.413 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.405+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc029b00 con 0x7f21cc0138f0 2021-03-09T10:53:55.414 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.409+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15236 cs=0 l=1 rev1=1 rx=0x7f21e0009140 tx=0x7f21e0004070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:55.416 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:53:55.413+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:53:55.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0015160 con 0x7f21cc0138f0 2021-03-09T10:53:55.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0007de0 con 0x7f21cc0138f0 2021-03-09T10:53:55.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.417+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:53:55.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.417+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:53:55.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.417+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:53:55.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:53:55.417+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:53:59.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:53:59.451 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:00.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:00.417 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:00.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15236 cs=0 l=1 rev1=1 rx=0x7f21e0009140 tx=0x7f21e0004070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:00.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15236 cs=0 l=1 rev1=1 rx=0x7f21e0009140 tx=0x7f21e0004070).stop 2021-03-09T10:54:00.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:00.418 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:00.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:00.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:00.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:00.419 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03d660 con 0x7f21c400a6b0 2021-03-09T10:54:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:54:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c4004620 2021-03-09T10:54:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:00.420 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:00.421 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.413+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:00.422 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d110 con 0x7f21c4004620 2021-03-09T10:54:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00f1b0 con 0x7f21c4004620 2021-03-09T10:54:00.423 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.417+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15246 cs=0 l=1 rev1=1 rx=0x7f21cc03e8e0 tx=0x7f21dc015b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:00.426 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:00.421+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52592 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:00.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.421+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:54:00.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:54:00.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21c4004620 2021-03-09T10:54:00.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.425+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:00.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.425+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:00.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:00.425+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:05.427 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.421+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15246 cs=0 l=1 rev1=1 rx=0x7f21cc03e8e0 tx=0x7f21dc015b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15246 cs=0 l=1 rev1=1 rx=0x7f21cc03e8e0 tx=0x7f21dc015b00).stop 2021-03-09T10:54:05.428 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:05.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:05.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:05.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:05.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:05.429 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00d2a0 con 0x7f21c400a6b0 2021-03-09T10:54:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029b00 con 0x7f21c40037a0 2021-03-09T10:54:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a850 con 0x7f21cc0138f0 2021-03-09T10:54:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:05.430 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:05.431 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:05.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:05.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:05.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:05.432 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:05.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:05.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:05.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01a990 con 0x7f21cc0138f0 2021-03-09T10:54:05.433 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.425+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03d7f0 con 0x7f21cc0138f0 2021-03-09T10:54:05.434 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.429+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15261 cs=0 l=1 rev1=1 rx=0x7f21e0033dc0 tx=0x7f21e0059730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:05.436 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:05.433+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52629 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:05.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:54:05.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.437+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:54:05.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0014230 con 0x7f21cc0138f0 2021-03-09T10:54:05.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.441+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:05.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.441+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:05.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:05.441+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:10.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:10.437 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:10.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15261 cs=0 l=1 rev1=1 rx=0x7f21e0033dc0 tx=0x7f21e0059730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:10.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15261 cs=0 l=1 rev1=1 rx=0x7f21e0033dc0 tx=0x7f21e0059730).stop 2021-03-09T10:54:10.438 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:10.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:10.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:10.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:10.439 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:10.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e70 con 0x7f21c400a6b0 2021-03-09T10:54:10.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a850 con 0x7f21c40037a0 2021-03-09T10:54:10.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00a490 con 0x7f21c4004620 2021-03-09T10:54:10.440 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:10.441 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:10.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:10.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:10.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.433+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:10.442 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:10.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:10.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:10.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:10.443 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:54:10.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d110 con 0x7f21c4004620 2021-03-09T10:54:10.444 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.437+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15268 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc0831c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:10.445 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:10.441+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52668 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:10.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:54:10.446 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21c4004620 2021-03-09T10:54:10.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.445+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:54:10.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.445+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:10.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.445+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:10.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:10.445+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:15.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:15.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:15.447 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15268 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc0831c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:15.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15268 cs=0 l=1 rev1=1 rx=0x7f21dc080150 tx=0x7f21dc0831c0).stop 2021-03-09T10:54:15.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:15.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:15.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:15.448 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e70 con 0x7f21c400a6b0 2021-03-09T10:54:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057160 con 0x7f21c40037a0 2021-03-09T10:54:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cf0 con 0x7f21cc0138f0 2021-03-09T10:54:15.449 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:15.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:15.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.441+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:15.450 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:15.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:15.451 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:15.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:15.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:15.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:15.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0138f0 2021-03-09T10:54:15.452 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d110 con 0x7f21cc0138f0 2021-03-09T10:54:15.453 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.445+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15278 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc005230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:15.455 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:15.453+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52705 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:15.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089e40 con 0x7f21cc0138f0 2021-03-09T10:54:15.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc08a120 con 0x7f21cc0138f0 2021-03-09T10:54:15.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.457+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0138f0 2021-03-09T10:54:15.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.457+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:15.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.457+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:15.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:15.457+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:20.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:20.456 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:20.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15278 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc005230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:20.457 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15278 cs=0 l=1 rev1=1 rx=0x7f21dc07b040 tx=0x7f21dc005230).stop 2021-03-09T10:54:20.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:20.458 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:20.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:20.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:20.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:20.459 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e70 con 0x7f21c400a6b0 2021-03-09T10:54:20.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057300 con 0x7f21c40037a0 2021-03-09T10:54:20.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc018530 con 0x7f21c4004620 2021-03-09T10:54:20.460 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:20.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:20.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:20.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:20.461 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.453+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:20.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:20.462 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:20.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:20.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:20.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:54:20.463 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01a990 con 0x7f21c4004620 2021-03-09T10:54:20.464 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.457+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15295 cs=0 l=1 rev1=1 rx=0x7f21e003dc80 tx=0x7f21e0029110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:20.466 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:20.465+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52744 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:20.467 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0007de0 con 0x7f21c4004620 2021-03-09T10:54:20.468 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0033c20 con 0x7f21c4004620 2021-03-09T10:54:20.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:54:20.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.465+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:20.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:20.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:20.469+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:25.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:25.469 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:25.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15295 cs=0 l=1 rev1=1 rx=0x7f21e003dc80 tx=0x7f21e0029110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:25.470 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15295 cs=0 l=1 rev1=1 rx=0x7f21e003dc80 tx=0x7f21e0029110).stop 2021-03-09T10:54:25.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:25.471 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:25.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:25.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:25.472 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:25.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a250 con 0x7f21c400a6b0 2021-03-09T10:54:25.473 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc093e50 con 0x7f21c40037a0 2021-03-09T10:54:25.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:25.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cf0 con 0x7f21cc0138f0 2021-03-09T10:54:25.474 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:25.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:25.475 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:25.476 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:25.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:25.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:25.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:25.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:25.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0138f0 2021-03-09T10:54:25.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01a990 con 0x7f21cc0138f0 2021-03-09T10:54:25.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.465+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15300 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e0091ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:25.481 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:25.473+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52781 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:25.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0006b30 con 0x7f21cc0138f0 2021-03-09T10:54:25.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.473+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21cc0138f0 2021-03-09T10:54:25.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.477+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0033c20 con 0x7f21cc0138f0 2021-03-09T10:54:25.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.477+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:25.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:25.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:25.477+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:29.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:29.516 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:30.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:30.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:30.478 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15300 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e0091ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:30.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15300 cs=0 l=1 rev1=1 rx=0x7f21e0092040 tx=0x7f21e0091ef0).stop 2021-03-09T10:54:30.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:30.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:30.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:30.479 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:30.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:30.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cf0 con 0x7f21c400a6b0 2021-03-09T10:54:30.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a990 con 0x7f21c40037a0 2021-03-09T10:54:30.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:54:30.480 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:30.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:30.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:30.481 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:30.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:30.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:30.482 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:30.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:30.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:30.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057110 con 0x7f21c4004620 2021-03-09T10:54:30.483 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.476+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d110 con 0x7f21c4004620 2021-03-09T10:54:30.484 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.480+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15312 cs=0 l=1 rev1=1 rx=0x7f21cc029da0 tx=0x7f21dc02df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:30.486 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:30.484+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52820 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:30.487 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:54:30.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc010d20 con 0x7f21c4004620 2021-03-09T10:54:30.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.488+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4004620 2021-03-09T10:54:30.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.488+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:30.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:30.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:30.488+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:35.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:35.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:35.488 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15312 cs=0 l=1 rev1=1 rx=0x7f21cc029da0 tx=0x7f21dc02df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:35.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15312 cs=0 l=1 rev1=1 rx=0x7f21cc029da0 tx=0x7f21dc02df50).stop 2021-03-09T10:54:35.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:35.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:35.489 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:35.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:35.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:35.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0955e0 con 0x7f21c400a6b0 2021-03-09T10:54:35.490 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a990 con 0x7f21c40037a0 2021-03-09T10:54:35.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21cc0138f0 2021-03-09T10:54:35.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:35.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:35.491 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:35.492 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:35.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:35.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:35.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:35.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:35.493 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:35.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc057960 con 0x7f21cc0138f0 2021-03-09T10:54:35.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21cc0138f0 2021-03-09T10:54:35.494 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.484+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15321 cs=0 l=1 rev1=1 rx=0x7f21e0009190 tx=0x7f21e0007de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:35.496 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:35.492+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52857 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:35.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.492+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21cc0138f0 2021-03-09T10:54:35.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21cc0138f0 2021-03-09T10:54:35.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.500+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0138f0 2021-03-09T10:54:35.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.500+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:35.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:35.505 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:35.500+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:40.497 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:40.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:40.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15321 cs=0 l=1 rev1=1 rx=0x7f21e0009190 tx=0x7f21e0007de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:40.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15321 cs=0 l=1 rev1=1 rx=0x7f21e0009190 tx=0x7f21e0007de0).stop 2021-03-09T10:54:40.498 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:40.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:40.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:40.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:40.499 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:40.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a990 con 0x7f21c400a6b0 2021-03-09T10:54:40.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017cd0 con 0x7f21c40037a0 2021-03-09T10:54:40.500 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0955e0 con 0x7f21c4004620 2021-03-09T10:54:40.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:40.501 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:40.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:40.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:40.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:40.502 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:40.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:40.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:40.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:40.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:54:40.503 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041a20 con 0x7f21c4004620 2021-03-09T10:54:40.504 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.496+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15333 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4048630).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:40.505 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:40.500+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52895 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:40.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408a220 con 0x7f21c4004620 2021-03-09T10:54:40.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:54:40.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e408a220 con 0x7f21c4004620 2021-03-09T10:54:40.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.504+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:40.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:40.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:40.508+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:45.506 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:45.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:45.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15333 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4048630).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:45.507 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15333 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4048630).stop 2021-03-09T10:54:45.508 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:45.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:45.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:45.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:45.509 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:45.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0431f0 con 0x7f21c400a6b0 2021-03-09T10:54:45.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0421f0 con 0x7f21c40037a0 2021-03-09T10:54:45.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a990 con 0x7f21cc0138f0 2021-03-09T10:54:45.510 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:45.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:45.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:45.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:45.511 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:45.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:45.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:45.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:45.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:45.512 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21cc0138f0 2021-03-09T10:54:45.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc041a20 con 0x7f21cc0138f0 2021-03-09T10:54:45.513 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.504+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15342 cs=0 l=1 rev1=1 rx=0x7f21e4009510 tx=0x7f21e4003bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:45.515 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52933 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:45.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40a3350 con 0x7f21cc0138f0 2021-03-09T10:54:45.516 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40a34b0 con 0x7f21cc0138f0 2021-03-09T10:54:45.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40a1990 con 0x7f21cc0138f0 2021-03-09T10:54:45.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:45.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:45.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:45.512+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:50.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:50.517 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:50.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15342 cs=0 l=1 rev1=1 rx=0x7f21e4009510 tx=0x7f21e4003bd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:50.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=15342 cs=0 l=1 rev1=1 rx=0x7f21e4009510 tx=0x7f21e4003bd0).stop 2021-03-09T10:54:50.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 msgr2=0x7f21cc041530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:50.518 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=CLOSED pgs=15342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:50.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:50.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:50.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:50.519 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:54:50.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091780 con 0x7f21c40037a0 2021-03-09T10:54:50.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0918c0 con 0x7f21c4004620 2021-03-09T10:54:50.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:50.520 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:50.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:50.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:50.521 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.512+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:50.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:50.522 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:50.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc0392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:50.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc0392e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:50.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:54:50.523 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057960 con 0x7f21c4004620 2021-03-09T10:54:50.524 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.516+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15353 cs=0 l=1 rev1=1 rx=0x7f21e007c5b0 tx=0x7f21e0021180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:50.525 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:50.520+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52971 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:50.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.520+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0004d80 con 0x7f21c4004620 2021-03-09T10:54:50.529 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00204c0 con 0x7f21c4004620 2021-03-09T10:54:50.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.528+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:54:50.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.528+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:50.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.528+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:50.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:50.528+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:51.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 msgr2=0x7f21cc012490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:54:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 msgr2=0x7f21cc012490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 0x7f21cc012490 secure :-1 s=READY pgs=12731 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc056ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 0x7f21cc012490 secure :-1 s=READY pgs=12731 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc056ec0).stop 2021-03-09T10:54:51.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:54:51.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21ea7fc700 4 mgrc ms_handle_reset ms_handle_reset con 0x7f21c4005da0 2021-03-09T10:54:51.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21ea7fc700 4 mgrc reconnect Terminating session with v2:172.21.15.38:6808/14374 2021-03-09T10:54:51.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:51.998 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21c4005da0 0x7f21cc012490 unknown :-1 s=CLOSED pgs=12731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:51.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:51.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:51.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc041530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:52.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:51.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] conn(0x7f21cc0138f0 0x7f21cc041530 secure :-1 s=READY pgs=14540 cs=0 l=1 rev1=1 rx=0x7f21e0029d80 tx=0x7f21dc056ec0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:55.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:55.526 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:55.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15353 cs=0 l=1 rev1=1 rx=0x7f21e007c5b0 tx=0x7f21e0021180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:55.527 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 secure :-1 s=READY pgs=15353 cs=0 l=1 rev1=1 rx=0x7f21e007c5b0 tx=0x7f21e0021180).stop 2021-03-09T10:54:55.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc011540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:55.528 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc011540 unknown :-1 s=CLOSED pgs=15353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:55.530 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:55.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:55.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc011a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:55.531 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:54:55.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:54:55.532 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091920 con 0x7f21cc0392e0 2021-03-09T10:54:55.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:54:55.533 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:55.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:54:55.534 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:55.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc011a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:55.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:55.535 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:55.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:55.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:55.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21cc0392e0 2021-03-09T10:54:55.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057110 con 0x7f21cc0392e0 2021-03-09T10:54:55.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.524+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc011a30 secure :-1 s=READY pgs=15364 cs=0 l=1 rev1=1 rx=0x7f21cc017810 tx=0x7f21dc027000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:55.538 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:54:55.532+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53009 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:54:55.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0392e0 2021-03-09T10:54:55.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0392e0 2021-03-09T10:54:55.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.536+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069500 con 0x7f21cc0392e0 2021-03-09T10:54:55.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.536+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:54:55.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.536+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:54:55.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:54:55.536+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:54:59.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:54:59.569 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:00.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc011a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:00.536 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc011a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:00.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc011a30 secure :-1 s=READY pgs=15364 cs=0 l=1 rev1=1 rx=0x7f21cc017810 tx=0x7f21dc027000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:00.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc011a30 secure :-1 s=READY pgs=15364 cs=0 l=1 rev1=1 rx=0x7f21cc017810 tx=0x7f21dc027000).stop 2021-03-09T10:55:00.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc011a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:00.537 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc011a30 unknown :-1 s=CLOSED pgs=15364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:00.538 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:00.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:00.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:00.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:55:00.539 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:55:00.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046910 con 0x7f21c4004620 2021-03-09T10:55:00.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:00.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:00.540 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:00.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:00.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:00.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:00.541 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:00.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:00.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:00.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc041a20 con 0x7f21c4004620 2021-03-09T10:55:00.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:55:00.542 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.532+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091440 secure :-1 s=READY pgs=15374 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e0015570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:00.546 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:00.540+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53047 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:00.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0020120 con 0x7f21c4004620 2021-03-09T10:55:00.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4004620 2021-03-09T10:55:00.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.548+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:55:00.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.548+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:00.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.548+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:00.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:00.548+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:05.547 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:05.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:05.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091440 secure :-1 s=READY pgs=15374 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e0015570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:05.548 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091440 secure :-1 s=READY pgs=15374 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e0015570).stop 2021-03-09T10:55:05.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:05.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091440 unknown :-1 s=CLOSED pgs=15374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:05.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:05.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:05.549 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:05.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:55:05.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:55:05.551 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039a00 con 0x7f21cc0392e0 2021-03-09T10:55:05.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:05.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:05.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:05.552 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:05.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:05.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:05.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:05.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:05.553 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:05.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046a70 con 0x7f21cc0392e0 2021-03-09T10:55:05.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.544+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:55:05.554 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.548+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15385 cs=0 l=1 rev1=1 rx=0x7f21e00936e0 tx=0x7f21e007d5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:05.558 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:05.552+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53085 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:05.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0392e0 2021-03-09T10:55:05.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0392e0 2021-03-09T10:55:05.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0076d80 con 0x7f21cc0392e0 2021-03-09T10:55:05.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.556+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:05.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.556+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:05.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:05.556+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:10.559 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:10.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:10.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15385 cs=0 l=1 rev1=1 rx=0x7f21e00936e0 tx=0x7f21e007d5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:10.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15385 cs=0 l=1 rev1=1 rx=0x7f21e00936e0 tx=0x7f21e007d5b0).stop 2021-03-09T10:55:10.560 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:10.561 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:10.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:10.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0913e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:10.562 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:10.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:55:10.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:55:10.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043a60 con 0x7f21c4004620 2021-03-09T10:55:10.563 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:10.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:10.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:10.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:10.564 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0913e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:10.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:10.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:10.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:10.565 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:10.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0918d0 con 0x7f21c4004620 2021-03-09T10:55:10.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.556+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21c4004620 2021-03-09T10:55:10.566 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.560+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0913e0 secure :-1 s=READY pgs=15396 cs=0 l=1 rev1=1 rx=0x7f21e4048520 tx=0x7f21e407f260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:10.569 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:10.564+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53123 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:10.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4031360 con 0x7f21c4004620 2021-03-09T10:55:10.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.564+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408fc80 con 0x7f21c4004620 2021-03-09T10:55:10.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4030f50 con 0x7f21c4004620 2021-03-09T10:55:10.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.568+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:10.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.568+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:10.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:10.568+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:15.570 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0913e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:15.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0913e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:15.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0913e0 secure :-1 s=READY pgs=15396 cs=0 l=1 rev1=1 rx=0x7f21e4048520 tx=0x7f21e407f260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:15.571 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0913e0 secure :-1 s=READY pgs=15396 cs=0 l=1 rev1=1 rx=0x7f21e4048520 tx=0x7f21e407f260).stop 2021-03-09T10:55:15.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0913e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:15.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0913e0 unknown :-1 s=CLOSED pgs=15396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:15.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:15.572 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:15.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:15.573 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:55:15.574 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:55:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017680 con 0x7f21cc0392e0 2021-03-09T10:55:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:15.575 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:15.576 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:15.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:15.580 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:15.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21cc0392e0 2021-03-09T10:55:15.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046a70 con 0x7f21cc0392e0 2021-03-09T10:55:15.581 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.568+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15406 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e000b490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:15.582 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:15.576+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53182 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:15.582 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007b230 con 0x7f21cc0392e0 2021-03-09T10:55:15.583 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.576+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007b230 con 0x7f21cc0392e0 2021-03-09T10:55:15.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e0076a20 con 0x7f21cc0392e0 2021-03-09T10:55:15.584 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.580+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:15.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:15.585 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:15.580+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:20.588 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:20.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:20.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15406 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e000b490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:20.589 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15406 cs=0 l=1 rev1=1 rx=0x7f21e00036c0 tx=0x7f21e000b490).stop 2021-03-09T10:55:20.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:20.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:20.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:20.590 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0911f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:55:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:55:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017680 con 0x7f21c4004620 2021-03-09T10:55:20.591 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:20.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:20.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:20.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:20.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0911f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:20.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:20.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:20.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:20.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:20.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc030da0 con 0x7f21c4004620 2021-03-09T10:55:20.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0918d0 con 0x7f21c4004620 2021-03-09T10:55:20.594 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.580+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0911f0 secure :-1 s=READY pgs=15417 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e4005210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:20.595 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:20.588+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53265 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:20.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.588+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4055390 con 0x7f21c4004620 2021-03-09T10:55:20.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.592+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40039a0 con 0x7f21c4004620 2021-03-09T10:55:20.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.596+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:55:20.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.596+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:20.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.596+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:20.600 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:20.596+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:25.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0911f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:25.592 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0911f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:25.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0911f0 secure :-1 s=READY pgs=15417 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e4005210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:25.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0911f0 secure :-1 s=READY pgs=15417 cs=0 l=1 rev1=1 rx=0x7f21e40052d0 tx=0x7f21e4005210).stop 2021-03-09T10:55:25.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc0911f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:25.593 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc0911f0 unknown :-1 s=CLOSED pgs=15417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:25.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:25.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:25.595 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:25.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a9e0 con 0x7f21c400a6b0 2021-03-09T10:55:25.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c40037a0 2021-03-09T10:55:25.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017680 con 0x7f21cc0392e0 2021-03-09T10:55:25.596 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:25.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:25.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:25.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:25.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:25.597 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:25.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:25.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:25.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:25.598 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc017cd0 con 0x7f21cc0392e0 2021-03-09T10:55:25.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc030da0 con 0x7f21cc0392e0 2021-03-09T10:55:25.599 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.591+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15428 cs=0 l=1 rev1=1 rx=0x7f21cc03ec70 tx=0x7f21e408fe20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:25.600 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:25.595+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53348 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:25.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4016850 con 0x7f21cc0392e0 2021-03-09T10:55:25.601 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.595+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4055a80 con 0x7f21cc0392e0 2021-03-09T10:55:25.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4055720 con 0x7f21cc0392e0 2021-03-09T10:55:25.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.599+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:25.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:25.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:25.599+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:29.692 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:29.701 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:30.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:30.602 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:30.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15428 cs=0 l=1 rev1=1 rx=0x7f21cc03ec70 tx=0x7f21e408fe20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:30.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15428 cs=0 l=1 rev1=1 rx=0x7f21cc03ec70 tx=0x7f21e408fe20).stop 2021-03-09T10:55:30.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:30.603 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:30.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:30.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:30.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:30.604 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017680 con 0x7f21c400a6b0 2021-03-09T10:55:30.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043a60 con 0x7f21c40037a0 2021-03-09T10:55:30.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094320 con 0x7f21c4004620 2021-03-09T10:55:30.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:30.605 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:30.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:30.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:30.606 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:30.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:30.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:30.607 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:30.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:30.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:55:30.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc057110 con 0x7f21c4004620 2021-03-09T10:55:30.608 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.599+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091170 secure :-1 s=READY pgs=15438 cs=0 l=1 rev1=1 rx=0x7f21dc07ccd0 tx=0x7f21dc08a200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:30.610 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:30.603+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53431 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:30.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.603+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0695d0 con 0x7f21c4004620 2021-03-09T10:55:30.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0050e0 con 0x7f21c4004620 2021-03-09T10:55:30.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.607+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c4004620 2021-03-09T10:55:30.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.607+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:30.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:30.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:30.607+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:35.611 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:35.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:35.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091170 secure :-1 s=READY pgs=15438 cs=0 l=1 rev1=1 rx=0x7f21dc07ccd0 tx=0x7f21dc08a200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:35.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091170 secure :-1 s=READY pgs=15438 cs=0 l=1 rev1=1 rx=0x7f21dc07ccd0 tx=0x7f21dc08a200).stop 2021-03-09T10:55:35.612 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc091170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:35.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc091170 unknown :-1 s=CLOSED pgs=15438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:35.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:35.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:35.613 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:35.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017680 con 0x7f21c400a6b0 2021-03-09T10:55:35.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094320 con 0x7f21c40037a0 2021-03-09T10:55:35.614 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:55:35.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:35.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:35.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:35.615 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:35.616 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:35.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:35.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:35.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:35.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:35.617 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc091660 con 0x7f21cc0392e0 2021-03-09T10:55:35.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:55:35.618 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.611+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023060 secure :-1 s=READY pgs=15450 cs=0 l=1 rev1=1 rx=0x7f21dc079000 tx=0x7f21dc0ab180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:35.621 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:35.619+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53514 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:35.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.623+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0bad40 con 0x7f21cc0392e0 2021-03-09T10:55:35.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.627+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc020270 con 0x7f21cc0392e0 2021-03-09T10:55:35.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.627+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc0bad40 con 0x7f21cc0392e0 2021-03-09T10:55:35.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.627+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:35.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.627+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:35.629 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:35.627+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:40.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc023060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:55:40.622 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc023060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:40.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023060 secure :-1 s=READY pgs=15450 cs=0 l=1 rev1=1 rx=0x7f21dc079000 tx=0x7f21dc0ab180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:40.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023060 secure :-1 s=READY pgs=15450 cs=0 l=1 rev1=1 rx=0x7f21dc079000 tx=0x7f21dc0ab180).stop 2021-03-09T10:55:40.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc023060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:40.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023060 unknown :-1 s=CLOSED pgs=15450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:40.623 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:40.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:40.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:40.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc094320 con 0x7f21c400a6b0 2021-03-09T10:55:40.624 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:40.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c40037a0 2021-03-09T10:55:40.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc017630 con 0x7f21c4004620 2021-03-09T10:55:40.625 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.619+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:40.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:40.626 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:40.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:40.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:40.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:40.627 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:40.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:40.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:55:40.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21c4004620 2021-03-09T10:55:40.628 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.623+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f80 secure :-1 s=READY pgs=15460 cs=0 l=1 rev1=1 rx=0x7f21e4005280 tx=0x7f21e4091190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:40.632 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:40.631+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53597 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:40.633 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4017e40 con 0x7f21c4004620 2021-03-09T10:55:40.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.635+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4017e40 con 0x7f21c4004620 2021-03-09T10:55:40.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:55:40.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.639+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:40.639 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.639+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:40.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:40.639+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:45.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:45.634 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:45.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f80 secure :-1 s=READY pgs=15460 cs=0 l=1 rev1=1 rx=0x7f21e4005280 tx=0x7f21e4091190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:45.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f80 secure :-1 s=READY pgs=15460 cs=0 l=1 rev1=1 rx=0x7f21e4005280 tx=0x7f21e4091190).stop 2021-03-09T10:55:45.635 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:45.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f80 unknown :-1 s=CLOSED pgs=15460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:45.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:45.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:45.636 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:45.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a180 con 0x7f21c400a6b0 2021-03-09T10:55:45.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08d590 con 0x7f21c40037a0 2021-03-09T10:55:45.637 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:55:45.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:45.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:45.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:45.638 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.631+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:45.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:45.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:45.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:45.640 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:45.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:45.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:55:45.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21cc0392e0 2021-03-09T10:55:45.641 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.635+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023010 secure :-1 s=READY pgs=15472 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003cda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:45.642 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:45.639+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:45.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0392e0 2021-03-09T10:55:45.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.643+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21cc0392e0 2021-03-09T10:55:45.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.643+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0392e0 2021-03-09T10:55:45.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.643+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:45.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.647+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:45.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:45.647+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:50.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc023010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:50.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc023010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:50.643 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023010 secure :-1 s=READY pgs=15472 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003cda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:50.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023010 secure :-1 s=READY pgs=15472 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e003cda0).stop 2021-03-09T10:55:50.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc023010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:50.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc023010 unknown :-1 s=CLOSED pgs=15472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:50.644 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:50.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:50.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:50.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a250 con 0x7f21c400a6b0 2021-03-09T10:55:50.645 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04ec40 con 0x7f21c40037a0 2021-03-09T10:55:50.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030f40 con 0x7f21c4004620 2021-03-09T10:55:50.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:50.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:50.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:50.646 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:50.647 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.639+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:50.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:50.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:50.648 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:50.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:50.649 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:55:50.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc017cd0 con 0x7f21c4004620 2021-03-09T10:55:50.650 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.643+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f10 secure :-1 s=READY pgs=15482 cs=0 l=1 rev1=1 rx=0x7f21cc026500 tx=0x7f21e4036d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:50.651 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:50.651+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53763 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:50.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:55:50.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.655+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:55:50.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:55:50.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.659+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:50.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.659+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:50.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:50.659+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:55.652 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:55.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:55.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f10 secure :-1 s=READY pgs=15482 cs=0 l=1 rev1=1 rx=0x7f21cc026500 tx=0x7f21e4036d80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:55.653 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f10 secure :-1 s=READY pgs=15482 cs=0 l=1 rev1=1 rx=0x7f21cc026500 tx=0x7f21e4036d80).stop 2021-03-09T10:55:55.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:55.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090f10 unknown :-1 s=CLOSED pgs=15482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:55.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:55.654 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc022fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:55.655 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:55.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04a250 con 0x7f21c400a6b0 2021-03-09T10:55:55.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc053500 con 0x7f21c40037a0 2021-03-09T10:55:55.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:55:55.656 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:55:55.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:55.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc022fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:55.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:55:55.657 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.651+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:55.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:55.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:55.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:55.658 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:55.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03a180 con 0x7f21cc0392e0 2021-03-09T10:55:55.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:55:55.659 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.655+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc022fc0 secure :-1 s=READY pgs=15494 cs=0 l=1 rev1=1 rx=0x7f21e4036f00 tx=0x7f21e4007f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:55.661 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:55:55.659+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53846 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:55:55.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007d20 con 0x7f21cc0392e0 2021-03-09T10:55:55.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.667+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408a220 con 0x7f21cc0392e0 2021-03-09T10:55:55.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.667+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4033a90 con 0x7f21cc0392e0 2021-03-09T10:55:55.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.667+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:55:55.670 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.667+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:55:55.671 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:55:55.667+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:55:59.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:55:59.749 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:00.662 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc022fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:00.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc022fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:00.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc022fc0 secure :-1 s=READY pgs=15494 cs=0 l=1 rev1=1 rx=0x7f21e4036f00 tx=0x7f21e4007f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:00.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc022fc0 secure :-1 s=READY pgs=15494 cs=0 l=1 rev1=1 rx=0x7f21e4036f00 tx=0x7f21e4007f70).stop 2021-03-09T10:56:00.663 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc022fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:00.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc022fc0 unknown :-1 s=CLOSED pgs=15494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:00.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:00.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:00.664 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:00.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc023650 con 0x7f21c400a6b0 2021-03-09T10:56:00.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c40037a0 2021-03-09T10:56:00.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:56:00.665 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:00.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:00.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:00.666 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.659+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:00.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:00.667 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:00.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:00.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:00.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:00.668 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0940c0 con 0x7f21c4004620 2021-03-09T10:56:00.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:56:00.669 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.663+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090bd0 secure :-1 s=READY pgs=15502 cs=0 l=1 rev1=1 rx=0x7f21e0013580 tx=0x7f21e0090f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:00.671 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:00.671+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53928 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:00.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00909c0 con 0x7f21c4004620 2021-03-09T10:56:00.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.675+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4004620 2021-03-09T10:56:00.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:56:00.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.679+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:00.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.679+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:00.680 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:00.679+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:05.672 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:05.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:05.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090bd0 secure :-1 s=READY pgs=15502 cs=0 l=1 rev1=1 rx=0x7f21e0013580 tx=0x7f21e0090f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:05.673 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090bd0 secure :-1 s=READY pgs=15502 cs=0 l=1 rev1=1 rx=0x7f21e0013580 tx=0x7f21e0090f40).stop 2021-03-09T10:56:05.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:05.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090bd0 unknown :-1 s=CLOSED pgs=15502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:05.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:05.674 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0910c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:05.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:05.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:05.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c400a6b0 2021-03-09T10:56:05.675 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:05.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21c40037a0 2021-03-09T10:56:05.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022fc0 con 0x7f21cc0392e0 2021-03-09T10:56:05.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:05.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:05.676 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.671+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:05.677 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:05.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:05.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:05.678 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:05.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc023100 con 0x7f21cc0392e0 2021-03-09T10:56:05.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03a180 con 0x7f21cc0392e0 2021-03-09T10:56:05.679 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.675+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0910c0 secure :-1 s=READY pgs=15518 cs=0 l=1 rev1=1 rx=0x7f21e4091160 tx=0x7f21e400f240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:05.680 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:05.679+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54003 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:05.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b43b0 con 0x7f21cc0392e0 2021-03-09T10:56:05.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.683+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e405faf0 con 0x7f21cc0392e0 2021-03-09T10:56:05.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.687+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4008ac0 con 0x7f21cc0392e0 2021-03-09T10:56:05.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.687+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:05.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.687+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:05.689 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:05.687+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:10.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:10.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:10.681 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0910c0 secure :-1 s=READY pgs=15518 cs=0 l=1 rev1=1 rx=0x7f21e4091160 tx=0x7f21e400f240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:10.682 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0910c0 secure :-1 s=READY pgs=15518 cs=0 l=1 rev1=1 rx=0x7f21e4091160 tx=0x7f21e400f240).stop 2021-03-09T10:56:10.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0910c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:10.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0910c0 unknown :-1 s=CLOSED pgs=15518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:10.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:10.683 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:10.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:10.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022fc0 con 0x7f21c400a6b0 2021-03-09T10:56:10.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c40037a0 2021-03-09T10:56:10.684 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a4c0 con 0x7f21c4004620 2021-03-09T10:56:10.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:10.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:10.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:10.685 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:10.686 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.679+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:10.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:10.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:10.687 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:10.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:10.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:56:10.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091660 con 0x7f21c4004620 2021-03-09T10:56:10.688 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.683+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090890 secure :-1 s=READY pgs=15524 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:10.692 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:10.691+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54085 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:10.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc083380 con 0x7f21c4004620 2021-03-09T10:56:10.692 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc021e50 con 0x7f21c4004620 2021-03-09T10:56:10.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:56:10.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.691+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:10.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.695+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:10.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:10.695+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:15.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:15.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:15.693 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090890 secure :-1 s=READY pgs=15524 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:15.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090890 secure :-1 s=READY pgs=15524 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc010e10).stop 2021-03-09T10:56:15.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:15.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090890 unknown :-1 s=CLOSED pgs=15524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:15.694 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:15.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:15.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:15.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc022fc0 con 0x7f21c400a6b0 2021-03-09T10:56:15.695 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc043350 con 0x7f21c40037a0 2021-03-09T10:56:15.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:15.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a660 con 0x7f21cc0392e0 2021-03-09T10:56:15.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:15.696 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:15.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:15.697 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:15.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:15.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:15.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:15.698 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:15.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:56:15.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.691+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc023100 con 0x7f21cc0392e0 2021-03-09T10:56:15.699 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.695+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15534 cs=0 l=1 rev1=1 rx=0x7f21e40b5590 tx=0x7f21e407f1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:15.702 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:15.699+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54168 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:15.702 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.699+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40093e0 con 0x7f21cc0392e0 2021-03-09T10:56:15.703 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40093e0 con 0x7f21cc0392e0 2021-03-09T10:56:15.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.703+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e405ed70 con 0x7f21cc0392e0 2021-03-09T10:56:15.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.703+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:15.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.703+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:15.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:15.703+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:20.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.698+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:20.704 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.698+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:20.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.698+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15534 cs=0 l=1 rev1=1 rx=0x7f21e40b5590 tx=0x7f21e407f1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:20.705 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.698+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15534 cs=0 l=1 rev1=1 rx=0x7f21e40b5590 tx=0x7f21e407f1e0).stop 2021-03-09T10:56:20.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:20.706 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:20.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:20.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:20.707 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:20.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042470 con 0x7f21c400a6b0 2021-03-09T10:56:20.708 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:56:20.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01d980 con 0x7f21c4004620 2021-03-09T10:56:20.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:20.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:20.709 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:20.710 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:20.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.702+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:20.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:20.711 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:20.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:20.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:20.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc03a660 con 0x7f21c4004620 2021-03-09T10:56:20.712 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc091660 con 0x7f21c4004620 2021-03-09T10:56:20.713 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.706+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 secure :-1 s=READY pgs=15550 cs=0 l=1 rev1=1 rx=0x7f21dc089fd0 tx=0x7f21dc06ae90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:20.713 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:20.710+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54252 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:20.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4004620 2021-03-09T10:56:20.718 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.714+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089dc0 con 0x7f21c4004620 2021-03-09T10:56:20.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4004620 2021-03-09T10:56:20.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.718+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:20.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.718+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:20.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:20.718+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:25.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc084fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:25.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc084fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:25.714 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 secure :-1 s=READY pgs=15550 cs=0 l=1 rev1=1 rx=0x7f21dc089fd0 tx=0x7f21dc06ae90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:25.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 secure :-1 s=READY pgs=15550 cs=0 l=1 rev1=1 rx=0x7f21dc089fd0 tx=0x7f21dc06ae90).stop 2021-03-09T10:56:25.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc084fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:25.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 unknown :-1 s=CLOSED pgs=15550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:25.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:25.715 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:25.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:25.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21c400a6b0 2021-03-09T10:56:25.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:56:25.716 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029970 con 0x7f21cc0392e0 2021-03-09T10:56:25.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:25.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:25.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:25.717 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:25.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.710+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:25.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:25.719 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:25.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:25.720 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:25.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc042470 con 0x7f21cc0392e0 2021-03-09T10:56:25.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:56:25.721 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.714+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15556 cs=0 l=1 rev1=1 rx=0x7f21e400f400 tx=0x7f21e404ffa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:25.722 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:25.718+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54335 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:25.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e409cc80 con 0x7f21cc0392e0 2021-03-09T10:56:25.723 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.718+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e409cde0 con 0x7f21cc0392e0 2021-03-09T10:56:25.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.726+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cc80 con 0x7f21cc0392e0 2021-03-09T10:56:25.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.726+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:25.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.726+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:25.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:25.726+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:29.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:29.796 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:30.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:30.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:30.724 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15556 cs=0 l=1 rev1=1 rx=0x7f21e400f400 tx=0x7f21e404ffa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:30.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15556 cs=0 l=1 rev1=1 rx=0x7f21e400f400 tx=0x7f21e404ffa0).stop 2021-03-09T10:56:30.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:30.725 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:30.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:30.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:30.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:30.726 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21c400a6b0 2021-03-09T10:56:30.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:56:30.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a140 con 0x7f21c4004620 2021-03-09T10:56:30.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:30.727 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:30.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:30.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:30.728 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:30.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:30.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:30.729 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:30.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:30.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc090700 con 0x7f21c4004620 2021-03-09T10:56:30.730 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0940c0 con 0x7f21c4004620 2021-03-09T10:56:30.731 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.722+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090210 secure :-1 s=READY pgs=15567 cs=0 l=1 rev1=1 rx=0x7f21e00a3fe0 tx=0x7f21e00037f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:30.731 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:30.730+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54418 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:30.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:56:30.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21c4004620 2021-03-09T10:56:30.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21c4004620 2021-03-09T10:56:30.740 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.738+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:30.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.738+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:30.741 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:30.738+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:35.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:35.733 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:35.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090210 secure :-1 s=READY pgs=15567 cs=0 l=1 rev1=1 rx=0x7f21e00a3fe0 tx=0x7f21e00037f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:35.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090210 secure :-1 s=READY pgs=15567 cs=0 l=1 rev1=1 rx=0x7f21e00a3fe0 tx=0x7f21e00037f0).stop 2021-03-09T10:56:35.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc090210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:35.734 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc090210 unknown :-1 s=CLOSED pgs=15567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:35.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:35.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:35.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:35.735 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21c400a6b0 2021-03-09T10:56:35.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b170 con 0x7f21c40037a0 2021-03-09T10:56:35.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0463e0 con 0x7f21cc0392e0 2021-03-09T10:56:35.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:35.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:35.736 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:35.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:35.737 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:35.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:35.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:35.738 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc00d510 con 0x7f21cc0392e0 2021-03-09T10:56:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.730+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc03a660 con 0x7f21cc0392e0 2021-03-09T10:56:35.739 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.734+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15577 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc015a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:35.741 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:35.738+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54501 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:35.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06b000 con 0x7f21cc0392e0 2021-03-09T10:56:35.742 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06b000 con 0x7f21cc0392e0 2021-03-09T10:56:35.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.742+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc0392e0 2021-03-09T10:56:35.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.742+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:35.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.742+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:35.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:35.742+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:40.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:56:40.743 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:40.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15577 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc015a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:40.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15577 cs=0 l=1 rev1=1 rx=0x7f21dc013460 tx=0x7f21dc015a20).stop 2021-03-09T10:56:40.744 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc08fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:40.745 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:40.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21c400a6b0 2021-03-09T10:56:40.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:56:40.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.738+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc00b170 con 0x7f21c4004620 2021-03-09T10:56:40.746 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:40.747 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc08fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:40.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:40.748 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:40.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:40.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:40.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:56:40.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc042470 con 0x7f21c4004620 2021-03-09T10:56:40.749 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.742+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc08fed0 secure :-1 s=READY pgs=15588 cs=0 l=1 rev1=1 rx=0x7f21e4017600 tx=0x7f21e4028210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:40.751 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:40.750+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54585 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009940 con 0x7f21c4004620 2021-03-09T10:56:40.752 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009aa0 con 0x7f21c4004620 2021-03-09T10:56:40.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4009300 con 0x7f21c4004620 2021-03-09T10:56:40.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.750+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:40.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:40.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:40.754+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:56:45.753 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:45.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc08fed0 secure :-1 s=READY pgs=15588 cs=0 l=1 rev1=1 rx=0x7f21e4017600 tx=0x7f21e4028210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:45.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc08fed0 secure :-1 s=READY pgs=15588 cs=0 l=1 rev1=1 rx=0x7f21e4017600 tx=0x7f21e4028210).stop 2021-03-09T10:56:45.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc08fed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:45.754 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc08fed0 unknown :-1 s=CLOSED pgs=15588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:45.755 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc057450 con 0x7f21c400a6b0 2021-03-09T10:56:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:56:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21cc0392e0 2021-03-09T10:56:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:45.756 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:45.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:45.757 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:45.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:45.758 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:45.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:45.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:45.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.750+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:45.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:56:45.759 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.754+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:56:45.760 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.754+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15599 cs=0 l=1 rev1=1 rx=0x7f21cc04ad30 tx=0x7f21e408f310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:45.763 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:45.762+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54668 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:45.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4041430 con 0x7f21cc0392e0 2021-03-09T10:56:45.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.766+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4041430 con 0x7f21cc0392e0 2021-03-09T10:56:45.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.770+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40af020 con 0x7f21cc0392e0 2021-03-09T10:56:45.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.770+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:45.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.770+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:45.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:45.770+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:50.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:50.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:50.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15599 cs=0 l=1 rev1=1 rx=0x7f21cc04ad30 tx=0x7f21e408f310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:50.765 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15599 cs=0 l=1 rev1=1 rx=0x7f21cc04ad30 tx=0x7f21e408f310).stop 2021-03-09T10:56:50.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:50.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:50.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:50.766 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:50.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:50.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c400a6b0 2021-03-09T10:56:50.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21c40037a0 2021-03-09T10:56:50.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0478b0 con 0x7f21c4004620 2021-03-09T10:56:50.767 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:50.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:50.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:50.768 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:50.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.762+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:50.770 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:56:50.771 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d510 con 0x7f21c4004620 2021-03-09T10:56:50.772 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.766+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046000 secure :-1 s=READY pgs=15609 cs=0 l=1 rev1=1 rx=0x7f21dc09a500 tx=0x7f21dc09a110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:50.774 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:50.770+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54751 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:50.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc069ae0 con 0x7f21c4004620 2021-03-09T10:56:50.775 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21c4004620 2021-03-09T10:56:50.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071ab0 con 0x7f21c4004620 2021-03-09T10:56:50.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.774+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:50.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.774+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:50.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:50.774+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:55.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:55.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:55.776 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046000 secure :-1 s=READY pgs=15609 cs=0 l=1 rev1=1 rx=0x7f21dc09a500 tx=0x7f21dc09a110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:55.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046000 secure :-1 s=READY pgs=15609 cs=0 l=1 rev1=1 rx=0x7f21dc09a500 tx=0x7f21dc09a110).stop 2021-03-09T10:56:55.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc046000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:55.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc046000 unknown :-1 s=CLOSED pgs=15609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:55.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:55.777 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:55.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:55.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c400a6b0 2021-03-09T10:56:55.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0428e0 con 0x7f21c40037a0 2021-03-09T10:56:55.778 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029970 con 0x7f21cc0392e0 2021-03-09T10:56:55.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:56:55.779 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:55.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:55.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:56:55.780 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.774+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:55.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:55.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:55.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:55.781 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:55.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc01abe0 con 0x7f21cc0392e0 2021-03-09T10:56:55.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc090700 con 0x7f21cc0392e0 2021-03-09T10:56:55.782 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.778+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 secure :-1 s=READY pgs=15622 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e007c5b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:55.787 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:56:55.786+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54833 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:56:55.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.786+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21cc0392e0 2021-03-09T10:56:55.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.790+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00140d0 con 0x7f21cc0392e0 2021-03-09T10:56:55.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.790+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21cc0392e0 2021-03-09T10:56:55.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.790+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:56:55.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.790+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:56:55.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:56:55.790+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:56:59.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:56:59.856 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:00.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:00.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:00.788 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 secure :-1 s=READY pgs=15622 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e007c5b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:00.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 secure :-1 s=READY pgs=15622 cs=0 l=1 rev1=1 rx=0x7f21e00a4010 tx=0x7f21e007c5b0).stop 2021-03-09T10:57:00.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc08fed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:00.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 unknown :-1 s=CLOSED pgs=15622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:00.789 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:00.790 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:00.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:00.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c400a6b0 2021-03-09T10:57:00.791 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc029970 con 0x7f21c40037a0 2021-03-09T10:57:00.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc030440 con 0x7f21c4004620 2021-03-09T10:57:00.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:00.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:00.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:00.792 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:00.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:00.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:00.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:00.793 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:00.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.786+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:00.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.790+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:57:00.794 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.790+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d510 con 0x7f21c4004620 2021-03-09T10:57:00.795 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.790+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045e10 secure :-1 s=READY pgs=15632 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc010d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:00.797 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:00.794+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54917 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:00.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.798+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007c20 con 0x7f21c4004620 2021-03-09T10:57:00.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.802+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007c20 con 0x7f21c4004620 2021-03-09T10:57:00.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.802+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21c4004620 2021-03-09T10:57:00.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.802+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:00.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.802+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:00.806 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:00.802+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:05.798 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:05.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:05.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045e10 secure :-1 s=READY pgs=15632 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc010d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:05.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045e10 secure :-1 s=READY pgs=15632 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc010d20).stop 2021-03-09T10:57:05.799 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:05.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045e10 unknown :-1 s=CLOSED pgs=15632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:05.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:05.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:05.800 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:05.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c400a6b0 2021-03-09T10:57:05.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc042020 con 0x7f21c40037a0 2021-03-09T10:57:05.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:57:05.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:05.801 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:05.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:05.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:05.802 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.794+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:05.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:05.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:05.803 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:05.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:05.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:57:05.804 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d510 con 0x7f21cc0392e0 2021-03-09T10:57:05.805 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.798+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 secure :-1 s=READY pgs=15641 cs=0 l=1 rev1=1 rx=0x7f21cc026280 tx=0x7f21dc0b3ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:05.806 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54999 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:05.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007a10 con 0x7f21cc0392e0 2021-03-09T10:57:05.807 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007a10 con 0x7f21cc0392e0 2021-03-09T10:57:05.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc004f80 con 0x7f21cc0392e0 2021-03-09T10:57:05.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:05.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:05.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:05.806+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:10.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:57:10.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:10.808 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 secure :-1 s=READY pgs=15641 cs=0 l=1 rev1=1 rx=0x7f21cc026280 tx=0x7f21dc0b3ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:10.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 secure :-1 s=READY pgs=15641 cs=0 l=1 rev1=1 rx=0x7f21cc026280 tx=0x7f21dc0b3ec0).stop 2021-03-09T10:57:10.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc08fed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:10.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc08fed0 unknown :-1 s=CLOSED pgs=15641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:10.809 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:10.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:10.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:10.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097b70 con 0x7f21c400a6b0 2021-03-09T10:57:10.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:57:10.810 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:57:10.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:10.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:10.811 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:10.812 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:10.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:10.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:10.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:10.813 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:10.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:10.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:57:10.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.805+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc01abe0 con 0x7f21c4004620 2021-03-09T10:57:10.814 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.809+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045c20 secure :-1 s=READY pgs=15654 cs=0 l=1 rev1=1 rx=0x7f21e0014240 tx=0x7f21e00037a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:10.816 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:10.813+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55083 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:10.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007d540 con 0x7f21c4004620 2021-03-09T10:57:10.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00910e0 con 0x7f21c4004620 2021-03-09T10:57:10.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.821+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a3d80 con 0x7f21c4004620 2021-03-09T10:57:10.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.821+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:10.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.821+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:10.825 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:10.821+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:15.817 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:15.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:15.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045c20 secure :-1 s=READY pgs=15654 cs=0 l=1 rev1=1 rx=0x7f21e0014240 tx=0x7f21e00037a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:15.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045c20 secure :-1 s=READY pgs=15654 cs=0 l=1 rev1=1 rx=0x7f21e0014240 tx=0x7f21e00037a0).stop 2021-03-09T10:57:15.818 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:15.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045c20 unknown :-1 s=CLOSED pgs=15654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:15.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc046110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:15.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:15.819 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:15.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c400a6b0 2021-03-09T10:57:15.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097cc0 con 0x7f21c40037a0 2021-03-09T10:57:15.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:57:15.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:15.820 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:15.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:15.821 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:15.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.813+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc046110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:15.822 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:15.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:15.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:15.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:15.823 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025c10 con 0x7f21cc0392e0 2021-03-09T10:57:15.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:57:15.824 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.817+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc046110 secure :-1 s=READY pgs=15664 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:15.827 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:15.825+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55165 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:15.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.825+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0392e0 2021-03-09T10:57:15.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.829+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007ca70 con 0x7f21cc0392e0 2021-03-09T10:57:15.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.829+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0392e0 2021-03-09T10:57:15.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.829+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:15.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.833+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:15.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:15.833+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:20.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc046110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:20.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc046110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:20.829 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc046110 secure :-1 s=READY pgs=15664 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d600).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:20.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc046110 secure :-1 s=READY pgs=15664 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e007d600).stop 2021-03-09T10:57:20.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc046110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:20.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc046110 unknown :-1 s=CLOSED pgs=15664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:20.830 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:20.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:20.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:20.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c400a6b0 2021-03-09T10:57:20.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21c40037a0 2021-03-09T10:57:20.831 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0215a0 con 0x7f21c4004620 2021-03-09T10:57:20.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:20.832 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:20.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:20.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:20.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.825+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:20.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.829+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:20.833 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.829+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:20.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.829+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:20.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.829+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:20.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.829+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc08fed0 con 0x7f21c4004620 2021-03-09T10:57:20.834 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.829+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc00d510 con 0x7f21c4004620 2021-03-09T10:57:20.835 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.833+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15676 cs=0 l=1 rev1=1 rx=0x7f21dc0b8230 tx=0x7f21dc0b3f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:20.838 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:20.833+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55245 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:20.839 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc089810 con 0x7f21c4004620 2021-03-09T10:57:20.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc089810 con 0x7f21c4004620 2021-03-09T10:57:20.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.841+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc007040 con 0x7f21c4004620 2021-03-09T10:57:20.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.841+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:20.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.841+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:20.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:20.841+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:25.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:25.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:25.840 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15676 cs=0 l=1 rev1=1 rx=0x7f21dc0b8230 tx=0x7f21dc0b3f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:25.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15676 cs=0 l=1 rev1=1 rx=0x7f21dc0b8230 tx=0x7f21dc0b3f30).stop 2021-03-09T10:57:25.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:25.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:25.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:25.841 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:25.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc090060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:25.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c780 con 0x7f21c400a6b0 2021-03-09T10:57:25.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc056ef0 con 0x7f21c40037a0 2021-03-09T10:57:25.842 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:25.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a890 con 0x7f21cc0392e0 2021-03-09T10:57:25.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:25.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:25.843 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:25.844 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc090060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:25.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:25.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:25.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:25.845 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:25.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:57:25.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.837+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc08fed0 con 0x7f21cc0392e0 2021-03-09T10:57:25.846 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.841+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc090060 secure :-1 s=READY pgs=15684 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc050020).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:25.850 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:25.845+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55327 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:25.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc05cda0 con 0x7f21cc0392e0 2021-03-09T10:57:25.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0392e0 2021-03-09T10:57:25.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc071280 con 0x7f21cc0392e0 2021-03-09T10:57:25.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.849+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:25.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.853+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:25.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:25.853+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:29.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:29.902 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:30.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc090060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:30.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc090060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:30.851 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc090060 secure :-1 s=READY pgs=15684 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc050020).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:30.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc090060 secure :-1 s=READY pgs=15684 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc050020).stop 2021-03-09T10:57:30.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc090060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:30.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc090060 unknown :-1 s=CLOSED pgs=15684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:30.852 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:30.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:30.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:30.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0237e0 con 0x7f21c400a6b0 2021-03-09T10:57:30.853 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08fed0 con 0x7f21c40037a0 2021-03-09T10:57:30.854 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c730 con 0x7f21c4004620 2021-03-09T10:57:30.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:30.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:30.855 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:30.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:30.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:30.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:30.856 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:30.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:30.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:30.857 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:57:30.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:57:30.858 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.849+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15694 cs=0 l=1 rev1=1 rx=0x7f21e40b11f0 tx=0x7f21e40039a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:30.860 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:30.857+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55411 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:30.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:57:30.861 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:57:30.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.861+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027d40 con 0x7f21c4004620 2021-03-09T10:57:30.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.861+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:30.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.861+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:30.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:30.861+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:35.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:35.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:35.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15694 cs=0 l=1 rev1=1 rx=0x7f21e40b11f0 tx=0x7f21e40039a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:35.862 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15694 cs=0 l=1 rev1=1 rx=0x7f21e40b11f0 tx=0x7f21e40039a0).stop 2021-03-09T10:57:35.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:35.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:35.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:35.863 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:35.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:35.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:35.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:35.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c400a6b0 2021-03-09T10:57:35.864 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01c6e0 con 0x7f21c40037a0 2021-03-09T10:57:35.865 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc08fed0 con 0x7f21cc0392e0 2021-03-09T10:57:35.866 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:35.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:35.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.857+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:35.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:35.867 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:35.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:35.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:35.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc090040 con 0x7f21cc0392e0 2021-03-09T10:57:35.868 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:57:35.869 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.861+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15706 cs=0 l=1 rev1=1 rx=0x7f21dc015ab0 tx=0x7f21dc010a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:35.871 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:35.869+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55493 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:35.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.869+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0326f0 con 0x7f21cc0392e0 2021-03-09T10:57:35.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.877+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0326f0 con 0x7f21cc0392e0 2021-03-09T10:57:35.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.877+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc010690 con 0x7f21cc0392e0 2021-03-09T10:57:35.880 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.877+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:35.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.877+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:35.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:35.877+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:40.871 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:40.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:40.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15706 cs=0 l=1 rev1=1 rx=0x7f21dc015ab0 tx=0x7f21dc010a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:40.872 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15706 cs=0 l=1 rev1=1 rx=0x7f21dc015ab0 tx=0x7f21dc010a20).stop 2021-03-09T10:57:40.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:40.874 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=CLOSED pgs=15706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:40.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:40.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:40.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:40.875 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046750 con 0x7f21c400a6b0 2021-03-09T10:57:40.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:57:40.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:57:40.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:40.876 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:40.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:40.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:40.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:40.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:40.877 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:40.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:40.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:40.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032310 con 0x7f21c4004620 2021-03-09T10:57:40.878 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.869+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084070 con 0x7f21c4004620 2021-03-09T10:57:40.879 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.873+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15716 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0004f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:40.881 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:40.877+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55577 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:40.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.877+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:57:40.881 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.877+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a6190 con 0x7f21c4004620 2021-03-09T10:57:40.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.881+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e003dde0 con 0x7f21c4004620 2021-03-09T10:57:40.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.881+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:40.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.881+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:40.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:40.881+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:45.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:45.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:45.882 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15716 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0004f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:45.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15716 cs=0 l=1 rev1=1 rx=0x7f21e0080160 tx=0x7f21e0004f30).stop 2021-03-09T10:57:45.883 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:45.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:45.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:45.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:45.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:45.884 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c400a6b0 2021-03-09T10:57:45.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c40037a0 2021-03-09T10:57:45.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0465b0 con 0x7f21cc0392e0 2021-03-09T10:57:45.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:45.885 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:45.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:45.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:45.886 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:45.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:45.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:45.887 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:45.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:45.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0466f0 con 0x7f21cc0392e0 2021-03-09T10:57:45.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032310 con 0x7f21cc0392e0 2021-03-09T10:57:45.888 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.881+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15727 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007c5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:45.890 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:45.889+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55659 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:45.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e007fbe0 con 0x7f21cc0392e0 2021-03-09T10:57:45.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007fbe0 con 0x7f21cc0392e0 2021-03-09T10:57:45.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.893+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e00a12b0 con 0x7f21cc0392e0 2021-03-09T10:57:45.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.893+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:45.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.897+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:45.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:45.897+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:50.892 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:50.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:50.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15727 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007c5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:50.893 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15727 cs=0 l=1 rev1=1 rx=0x7f21e00a6480 tx=0x7f21e007c5f0).stop 2021-03-09T10:57:50.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:50.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=CLOSED pgs=15727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:50.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:50.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:50.894 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:50.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046750 con 0x7f21c400a6b0 2021-03-09T10:57:50.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:57:50.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:57:50.895 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:50.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:50.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:50.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:50.896 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:50.897 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:50.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:50.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:50.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:50.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc025ad0 con 0x7f21c4004620 2021-03-09T10:57:50.898 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.889+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:57:50.899 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.893+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15739 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e4028220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:50.901 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:50.897+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55743 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:50.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4004620 2021-03-09T10:57:50.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e408f000 con 0x7f21c4004620 2021-03-09T10:57:50.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.905+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4350 con 0x7f21c4004620 2021-03-09T10:57:50.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.905+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:50.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.905+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:50.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:50.905+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:55.902 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:57:55.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:55.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15739 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e4028220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:55.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15739 cs=0 l=1 rev1=1 rx=0x7f21e40282e0 tx=0x7f21e4028220).stop 2021-03-09T10:57:55.903 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:55.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:55.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:55.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:55.904 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:57:55.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:55.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf60 con 0x7f21c400a6b0 2021-03-09T10:57:55.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:57:55.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:57:55.905 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:55.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:57:55.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:55.906 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:55.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:55.907 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:57:55.908 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc090040 con 0x7f21cc0392e0 2021-03-09T10:57:55.909 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.901+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15748 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc021560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:55.913 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:57:55.909+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55825 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:57:55.913 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007110 con 0x7f21cc0392e0 2021-03-09T10:57:55.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007110 con 0x7f21cc0392e0 2021-03-09T10:57:55.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.913+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc062120 con 0x7f21cc0392e0 2021-03-09T10:57:55.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.913+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:57:55.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.913+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:57:55.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:57:55.913+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:57:59.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:57:59.948 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15748 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc021560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:00.914 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 secure :-1 s=READY pgs=15748 cs=0 l=1 rev1=1 rx=0x7f21dc006fd0 tx=0x7f21dc021560).stop 2021-03-09T10:58:00.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc0460c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:00.915 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc0460c0 unknown :-1 s=CLOSED pgs=15748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:00.916 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc046750 con 0x7f21c400a6b0 2021-03-09T10:58:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:58:00.917 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01cf10 con 0x7f21c4004620 2021-03-09T10:58:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:00.918 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.909+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:00.919 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:00.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:00.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:58:00.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0084080 con 0x7f21c4004620 2021-03-09T10:58:00.920 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.913+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15758 cs=0 l=1 rev1=1 rx=0x7f21e0014000 tx=0x7f21e0076f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:00.924 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:00.921+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55909 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:00.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:58:00.925 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.921+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:58:00.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.925+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:58:00.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.925+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:00.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.925+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:00.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:00.925+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:05.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:05.926 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:05.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15758 cs=0 l=1 rev1=1 rx=0x7f21e0014000 tx=0x7f21e0076f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:05.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15758 cs=0 l=1 rev1=1 rx=0x7f21e0014000 tx=0x7f21e0076f90).stop 2021-03-09T10:58:05.927 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:05.928 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:05.929 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:05.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:05.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:05.930 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026230 con 0x7f21c400a6b0 2021-03-09T10:58:05.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:58:05.931 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0323d0 con 0x7f21cc0392e0 2021-03-09T10:58:05.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:05.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:05.932 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:05.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:05.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:05.933 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:05.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:05.934 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:05.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:05.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc0460c0 con 0x7f21cc0392e0 2021-03-09T10:58:05.935 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:58:05.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.924+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031ee0 secure :-1 s=READY pgs=15769 cs=0 l=1 rev1=1 rx=0x7f21cc08d860 tx=0x7f21dc0b3fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:05.936 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:05.928+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55991 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:05.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc050880 con 0x7f21cc0392e0 2021-03-09T10:58:05.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc050ec0 con 0x7f21cc0392e0 2021-03-09T10:58:05.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.936+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0392e0 2021-03-09T10:58:05.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.936+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:05.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.936+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:05.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:05.936+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:10.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc031ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:10.936 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc031ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:10.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031ee0 secure :-1 s=READY pgs=15769 cs=0 l=1 rev1=1 rx=0x7f21cc08d860 tx=0x7f21dc0b3fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:10.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031ee0 secure :-1 s=READY pgs=15769 cs=0 l=1 rev1=1 rx=0x7f21cc08d860 tx=0x7f21dc0b3fd0).stop 2021-03-09T10:58:10.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc031ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:10.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031ee0 unknown :-1 s=CLOSED pgs=15769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:10.937 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:10.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:10.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:10.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011e90 con 0x7f21c400a6b0 2021-03-09T10:58:10.938 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:58:10.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c4004620 2021-03-09T10:58:10.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:10.939 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:10.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:10.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:10.940 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:10.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:10.941 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:10.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:10.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:10.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc02a260 con 0x7f21c4004620 2021-03-09T10:58:10.942 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:58:10.943 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.932+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15780 cs=0 l=1 rev1=1 rx=0x7f21e0020510 tx=0x7f21e0021f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:10.945 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:10.940+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56073 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:10.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e003c000 con 0x7f21c4004620 2021-03-09T10:58:10.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.940+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0020120 con 0x7f21c4004620 2021-03-09T10:58:10.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:58:10.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.944+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:10.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.944+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:10.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:10.944+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:15.946 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:15.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:15.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15780 cs=0 l=1 rev1=1 rx=0x7f21e0020510 tx=0x7f21e0021f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:15.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15780 cs=0 l=1 rev1=1 rx=0x7f21e0020510 tx=0x7f21e0021f90).stop 2021-03-09T10:58:15.947 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:15.948 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c400a6b0 2021-03-09T10:58:15.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:15.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032420 con 0x7f21c40037a0 2021-03-09T10:58:15.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc011e90 con 0x7f21cc0392e0 2021-03-09T10:58:15.949 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:15.950 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:15.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:15.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:15.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:15.951 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:15.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:15.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:15.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:58:15.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc025ad0 con 0x7f21cc0392e0 2021-03-09T10:58:15.952 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.944+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031cf0 secure :-1 s=READY pgs=15790 cs=0 l=1 rev1=1 rx=0x7f21e4039060 tx=0x7f21e4091210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:15.956 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:15.952+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56157 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:15.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0392e0 2021-03-09T10:58:15.957 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.952+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40099f0 con 0x7f21cc0392e0 2021-03-09T10:58:15.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21cc0392e0 2021-03-09T10:58:15.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.956+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:15.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.956+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:15.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:15.956+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:20.958 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc031cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:20.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc031cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:20.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031cf0 secure :-1 s=READY pgs=15790 cs=0 l=1 rev1=1 rx=0x7f21e4039060 tx=0x7f21e4091210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:20.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031cf0 secure :-1 s=READY pgs=15790 cs=0 l=1 rev1=1 rx=0x7f21e4039060 tx=0x7f21e4091210).stop 2021-03-09T10:58:20.959 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc031cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:20.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc031cf0 unknown :-1 s=CLOSED pgs=15790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:20.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:20.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:20.960 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:20.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c400a6b0 2021-03-09T10:58:20.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:58:20.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035bc0 con 0x7f21c4004620 2021-03-09T10:58:20.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:20.961 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:20.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:20.962 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:20.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:20.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:20.963 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:20.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:20.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:20.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc011e70 con 0x7f21c4004620 2021-03-09T10:58:20.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:58:20.964 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.956+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15801 cs=0 l=1 rev1=1 rx=0x7f21e4036560 tx=0x7f21e4036480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:20.967 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:20.964+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56239 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:20.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e408f960 con 0x7f21c4004620 2021-03-09T10:58:20.968 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:58:20.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:58:20.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.964+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:20.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:20.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:20.968+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:25.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:25.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:25.969 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15801 cs=0 l=1 rev1=1 rx=0x7f21e4036560 tx=0x7f21e4036480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:25.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 secure :-1 s=READY pgs=15801 cs=0 l=1 rev1=1 rx=0x7f21e4036560 tx=0x7f21e4036480).stop 2021-03-09T10:58:25.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc045bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:25.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc045bd0 unknown :-1 s=CLOSED pgs=15801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:25.970 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:25.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:25.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:25.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a590 con 0x7f21c400a6b0 2021-03-09T10:58:25.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc032240 con 0x7f21c40037a0 2021-03-09T10:58:25.971 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21cc0392e0 2021-03-09T10:58:25.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:25.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:25.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:25.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:25.972 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:25.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:25.973 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:25.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:25.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:25.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:58:25.974 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.964+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0460c0 con 0x7f21cc0392e0 2021-03-09T10:58:25.975 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.968+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15812 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc009850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:25.976 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:25.972+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56323 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:25.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ab2a0 con 0x7f21cc0392e0 2021-03-09T10:58:25.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.972+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ab0d0 con 0x7f21cc0392e0 2021-03-09T10:58:25.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0392e0 2021-03-09T10:58:25.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.976+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:25.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.976+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:25.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:25.976+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:29.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:29.994 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:58:30.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:30.977 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:30.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15812 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc009850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:30.978 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15812 cs=0 l=1 rev1=1 rx=0x7f21dc062000 tx=0x7f21dc009850).stop 2021-03-09T10:58:30.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:30.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:30.979 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:30.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:30.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:30.980 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:30.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c400a6b0 2021-03-09T10:58:30.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:30.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b90 con 0x7f21c40037a0 2021-03-09T10:58:30.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03a5e0 con 0x7f21c4004620 2021-03-09T10:58:30.981 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:30.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:30.982 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:30.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:30.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:30.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:30.983 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:30.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032240 con 0x7f21c4004620 2021-03-09T10:58:30.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc011e70 con 0x7f21c4004620 2021-03-09T10:58:30.984 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.976+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 secure :-1 s=READY pgs=15823 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e407d160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:30.988 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:30.984+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56404 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:30.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4004620 2021-03-09T10:58:30.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009230 con 0x7f21c4004620 2021-03-09T10:58:30.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.992+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e409cc90 con 0x7f21c4004620 2021-03-09T10:58:30.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.992+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:30.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.992+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:30.997 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:30.992+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:35.989 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc084fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:35.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc084fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:35.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 secure :-1 s=READY pgs=15823 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e407d160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:35.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 secure :-1 s=READY pgs=15823 cs=0 l=1 rev1=1 rx=0x7f21e40aa4e0 tx=0x7f21e407d160).stop 2021-03-09T10:58:35.990 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc084fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:35.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc084fe0 unknown :-1 s=CLOSED pgs=15823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:35.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:35.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:35.991 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:35.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b90 con 0x7f21c400a6b0 2021-03-09T10:58:35.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c40037a0 2021-03-09T10:58:35.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091b80 con 0x7f21cc0392e0 2021-03-09T10:58:35.992 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:35.993 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:35.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:35.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:35.994 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.984+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:35.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:35.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:35.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:35.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:35.995 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc046260 con 0x7f21cc0392e0 2021-03-09T10:58:35.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:58:35.996 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.988+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15837 cs=0 l=1 rev1=1 rx=0x7f21cc091360 tx=0x7f21dc09adb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:35.998 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:35.996+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56480 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:35.999 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:35.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc06a230 con 0x7f21cc0392e0 2021-03-09T10:58:36.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:36.000+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc06a230 con 0x7f21cc0392e0 2021-03-09T10:58:36.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:36.000+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc007040 con 0x7f21cc0392e0 2021-03-09T10:58:36.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:36.000+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:36.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:36.000+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:36.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:36.000+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:41.000 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:41.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:41.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15837 cs=0 l=1 rev1=1 rx=0x7f21cc091360 tx=0x7f21dc09adb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:41.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15837 cs=0 l=1 rev1=1 rx=0x7f21cc091360 tx=0x7f21dc09adb0).stop 2021-03-09T10:58:41.001 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:41.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:41.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:41.002 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:41.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:41.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b90 con 0x7f21c400a6b0 2021-03-09T10:58:41.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc02a120 con 0x7f21c40037a0 2021-03-09T10:58:41.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc091b60 con 0x7f21c4004620 2021-03-09T10:58:41.003 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:41.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:41.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:41.004 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:41.005 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:40.996+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:41.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:41.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:41.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:41.006 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:41.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc056660 con 0x7f21c4004620 2021-03-09T10:58:41.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc02a260 con 0x7f21c4004620 2021-03-09T10:58:41.007 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.000+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052d60 secure :-1 s=READY pgs=15844 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0091150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:41.008 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:41.004+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56562 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:41.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:58:41.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.004+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0075bb0 con 0x7f21c4004620 2021-03-09T10:58:41.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:58:41.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.008+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:41.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.008+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:41.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:41.008+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:46.009 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:46.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:46.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052d60 secure :-1 s=READY pgs=15844 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0091150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:46.010 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052d60 secure :-1 s=READY pgs=15844 cs=0 l=1 rev1=1 rx=0x7f21e007d880 tx=0x7f21e0091150).stop 2021-03-09T10:58:46.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:46.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052d60 unknown :-1 s=CLOSED pgs=15844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:46.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:46.011 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:46.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:46.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:46.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b60 con 0x7f21c400a6b0 2021-03-09T10:58:46.012 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026af0 con 0x7f21c40037a0 2021-03-09T10:58:46.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:46.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21cc0392e0 2021-03-09T10:58:46.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:46.013 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:46.014 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:46.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:46.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:46.015 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:46.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:46.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:58:46.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032240 con 0x7f21cc0392e0 2021-03-09T10:58:46.016 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.008+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029f80 secure :-1 s=READY pgs=15854 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e4003a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:46.018 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56645 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:46.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40b4cf0 con 0x7f21cc0392e0 2021-03-09T10:58:46.019 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40b4cf0 con 0x7f21cc0392e0 2021-03-09T10:58:46.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4036170 con 0x7f21cc0392e0 2021-03-09T10:58:46.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:46.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:46.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:46.016+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:51.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc029f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:51.020 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc029f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:51.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029f80 secure :-1 s=READY pgs=15854 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e4003a10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:51.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029f80 secure :-1 s=READY pgs=15854 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e4003a10).stop 2021-03-09T10:58:51.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc029f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:51.021 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029f80 unknown :-1 s=CLOSED pgs=15854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:51.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:51.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:51.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:51.022 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b60 con 0x7f21c400a6b0 2021-03-09T10:58:51.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc026af0 con 0x7f21c40037a0 2021-03-09T10:58:51.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c4004620 2021-03-09T10:58:51.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:51.023 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:51.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:51.024 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:51.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:51.025 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:51.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:51.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:51.026 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:51.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032240 con 0x7f21c4004620 2021-03-09T10:58:51.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc056660 con 0x7f21c4004620 2021-03-09T10:58:51.027 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.020+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052cf0 secure :-1 s=READY pgs=15867 cs=0 l=1 rev1=1 rx=0x7f21e009ae50 tx=0x7f21e007cb60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:51.031 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:51.028+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56724 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:51.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00138f0 con 0x7f21c4004620 2021-03-09T10:58:51.032 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.028+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e007c530 con 0x7f21c4004620 2021-03-09T10:58:51.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.032+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e007fb10 con 0x7f21c4004620 2021-03-09T10:58:51.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.032+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:51.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:51.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:51.032+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:58:56.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:56.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:56.033 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052cf0 secure :-1 s=READY pgs=15867 cs=0 l=1 rev1=1 rx=0x7f21e009ae50 tx=0x7f21e007cb60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:56.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052cf0 secure :-1 s=READY pgs=15867 cs=0 l=1 rev1=1 rx=0x7f21e009ae50 tx=0x7f21e007cb60).stop 2021-03-09T10:58:56.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:56.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052cf0 unknown :-1 s=CLOSED pgs=15867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:56.034 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:56.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:56.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:56.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b60 con 0x7f21c400a6b0 2021-03-09T10:58:56.035 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:58:56.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035bc0 con 0x7f21cc0392e0 2021-03-09T10:58:56.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:58:56.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:56.036 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:56.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:58:56.037 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:56.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:56.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:56.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:56.038 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:56.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032240 con 0x7f21cc0392e0 2021-03-09T10:58:56.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21cc0392e0 2021-03-09T10:58:56.039 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.031+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029ee0 secure :-1 s=READY pgs=15876 cs=0 l=1 rev1=1 rx=0x7f21cc0467b0 tx=0x7f21e4028200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:56.041 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:58:56.035+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:58:56.041 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40099f0 con 0x7f21cc0392e0 2021-03-09T10:58:56.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e40099f0 con 0x7f21cc0392e0 2021-03-09T10:58:56.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.039+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4cb0 con 0x7f21cc0392e0 2021-03-09T10:58:56.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.039+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:58:56.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.039+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:58:56.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:58:56.039+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:59:00.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:59:00.070 DEBUG:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-03-09T10:59:01.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc029ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:59:01.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc029ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:01.042 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029ee0 secure :-1 s=READY pgs=15876 cs=0 l=1 rev1=1 rx=0x7f21cc0467b0 tx=0x7f21e4028200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:01.043 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029ee0 secure :-1 s=READY pgs=15876 cs=0 l=1 rev1=1 rx=0x7f21cc0467b0 tx=0x7f21e4028200).stop 2021-03-09T10:59:01.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc029ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:01.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc029ee0 unknown :-1 s=CLOSED pgs=15876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:01.044 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:01.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:01.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:01.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b60 con 0x7f21c400a6b0 2021-03-09T10:59:01.045 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c40037a0 2021-03-09T10:59:01.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc035bc0 con 0x7f21c4004620 2021-03-09T10:59:01.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:59:01.046 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:01.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:59:01.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.035+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:01.047 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:01.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:01.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:01.048 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:01.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:01.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:59:01.049 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.039+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc032240 con 0x7f21c4004620 2021-03-09T10:59:01.050 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.043+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c70 secure :-1 s=READY pgs=15892 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e4005f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:01.052 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:59:01.047+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56889 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:59:01.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400dee0 con 0x7f21c4004620 2021-03-09T10:59:01.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.047+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009730 con 0x7f21c4004620 2021-03-09T10:59:01.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e4027bd0 con 0x7f21c4004620 2021-03-09T10:59:01.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.051+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:59:01.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.051+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:59:01.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:01.051+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:59:06.053 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:59:06.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:06.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c70 secure :-1 s=READY pgs=15892 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e4005f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:06.054 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c70 secure :-1 s=READY pgs=15892 cs=0 l=1 rev1=1 rx=0x7f21e407d330 tx=0x7f21e4005f20).stop 2021-03-09T10:59:06.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 msgr2=0x7f21cc052c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:06.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c70 unknown :-1 s=CLOSED pgs=15892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:06.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:06.055 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:06.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:06.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b60 con 0x7f21c400a6b0 2021-03-09T10:59:06.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc04c4c0 con 0x7f21c40037a0 2021-03-09T10:59:06.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc03e2a0 con 0x7f21cc0392e0 2021-03-09T10:59:06.056 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:59:06.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:06.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:59:06.057 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:06.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.051+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:06.058 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:06.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:06.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:06.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:06.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc032240 con 0x7f21cc0392e0 2021-03-09T10:59:06.059 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc046260 con 0x7f21cc0392e0 2021-03-09T10:59:06.060 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.055+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15899 cs=0 l=1 rev1=1 rx=0x7f21dc06a1c0 tx=0x7f21dc009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:06.062 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:59:06.063+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56974 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:59:06.063 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc032a50 con 0x7f21cc0392e0 2021-03-09T10:59:06.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0535a0 con 0x7f21cc0392e0 2021-03-09T10:59:06.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21dc069a10 con 0x7f21cc0392e0 2021-03-09T10:59:06.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.063+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:59:06.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.067+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:59:06.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:06.067+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:59:11.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:59:11.064 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:11.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15899 cs=0 l=1 rev1=1 rx=0x7f21dc06a1c0 tx=0x7f21dc009290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:11.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 secure :-1 s=READY pgs=15899 cs=0 l=1 rev1=1 rx=0x7f21dc06a1c0 tx=0x7f21dc009290).stop 2021-03-09T10:59:11.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 msgr2=0x7f21cc084af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:11.065 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21cc0392e0 0x7f21cc084af0 unknown :-1 s=CLOSED pgs=15899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:11.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:11.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:11.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:11.066 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc039b60 con 0x7f21c400a6b0 2021-03-09T10:59:11.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc097370 con 0x7f21c40037a0 2021-03-09T10:59:11.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc01a6b0 con 0x7f21c4004620 2021-03-09T10:59:11.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f1828700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.41:3300/0 2021-03-09T10:59:11.067 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f1828700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:11.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f2029700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:3300/0 2021-03-09T10:59:11.068 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f2029700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:11.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.063+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:11.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 msgr2=0x7f21cc016540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:11.069 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] conn(0x7f21c40037a0 0x7f21cc016540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:11.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 msgr2=0x7f21cc012460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:11.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21c400a6b0 0x7f21cc012460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:11.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f21cc090e00 con 0x7f21c4004620 2021-03-09T10:59:11.070 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 -- 172.21.15.38:0/3657190860 --> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc021460 con 0x7f21c4004620 2021-03-09T10:59:11.071 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.067+0000 7f21f282a700 1 --2- 172.21.15.38:0/3657190860 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f21c4004620 0x7f21cc052c00 secure :-1 s=READY pgs=15912 cs=0 l=1 rev1=1 rx=0x7f21e40038a0 tx=0x7f21e4091190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:11.072 INFO:tasks.ceph.mon.c.smithi038.stderr:2021-03-09T10:59:11.071+0000 7f10d79ec700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57056 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:54:23.264593+0000) 2021-03-09T10:59:11.073 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.071+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400da80 con 0x7f21c4004620 2021-03-09T10:59:11.074 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.075+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400da80 con 0x7f21c4004620 2021-03-09T10:59:11.079 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.079+0000 7f21ea7fc700 1 -- 172.21.15.38:0/3657190860 <== mon.2 v2:172.21.15.38:3301/0 3 ==== mgrmap(e 4) v1 ==== 103969+0+0 (secure 0 0 0) 0x7f21e40b4cb0 con 0x7f21c4004620 2021-03-09T10:59:11.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.079+0000 7f21ea7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2021-03-09T10:59:11.080 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.079+0000 7f21ea7fc700 4 mgrc handle_mgr_map Got map version 4 2021-03-09T10:59:11.081 INFO:teuthology.misc.health.smithi038.stderr:2021-03-09T10:59:11.079+0000 7f21ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.38:6808/14374,v1:172.21.15.38:6809/14374] 2021-03-09T10:59:15.753 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-09T10:59:15.754 DEBUG:teuthology.task.console_log:Killing console logger for smithi038 2021-03-09T10:59:15.755 DEBUG:teuthology.task.console_log:Killing console logger for smithi041 2021-03-09T10:59:15.756 DEBUG:teuthology.exit:Finished running handlers